[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:57 dist-test-slave-s6tm krb5kdc[14508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:57 dist-test-slave-s6tm krb5kdc[14508](info): set up 2 sockets
Sep 25 10:07:57 dist-test-slave-s6tm krb5kdc[14508](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 10:07:58.537719 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:07:58 dist-test-slave-s6tm krb5kdc[14508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:07:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:07:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:07:58.685036 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:43205
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:41933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:43205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:58.769050 14521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:58.769284 14521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:58.769308 14521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:58.770581 14521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:07:58.770632 14521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:58.770646 14521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:58.770659 14521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:07:58.770673 14521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:07:58.772315 14521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:41933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:43205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:43205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:58.772591 14521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:58.772854 14521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:07:58.778805 14521 server_base.cc:1034] running on GCE node
W20260925 10:07:58.778915 14527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:58.778831 14526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:58.778831 14529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:58.779376 14521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:58.779718 14521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:58.780934 14521 hybrid_clock.cc:648] HybridClock initialized: now 1790330878780979 us; error 98 us; skew 500 ppm
Sep 25 10:07:58 dist-test-slave-s6tm krb5kdc[14508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330878, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:58.783893 14521 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:07:58.784511 14521 webserver.cc:460] Webserver started at http://127.14.41.126:44511/ using document root <none> and password file <none>
I20260925 10:07:58.784761 14521 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:58.784832 14521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:58.784999 14521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:58.786267 14521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "6911b7c9f2ba4a1690e9bc6f43f29dbc"
format_stamp: "Formatted at 2026-09-25 10:07:58 on dist-test-slave-s6tm"
server_key: "3d3b6615bd626c6c7df96a18ec815689"
server_key_iv: "85c07e967addae15e65d9e29da6b54cb"
server_key_version: "encryptionkey@0"
I20260925 10:07:58.786680 14521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "6911b7c9f2ba4a1690e9bc6f43f29dbc"
format_stamp: "Formatted at 2026-09-25 10:07:58 on dist-test-slave-s6tm"
server_key: "3d3b6615bd626c6c7df96a18ec815689"
server_key_iv: "85c07e967addae15e65d9e29da6b54cb"
server_key_version: "encryptionkey@0"
I20260925 10:07:58.788295 14521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:07:58.789175 14536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:58.789525 14521 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:58.789671 14521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "6911b7c9f2ba4a1690e9bc6f43f29dbc"
format_stamp: "Formatted at 2026-09-25 10:07:58 on dist-test-slave-s6tm"
server_key: "3d3b6615bd626c6c7df96a18ec815689"
server_key_iv: "85c07e967addae15e65d9e29da6b54cb"
server_key_version: "encryptionkey@0"
I20260925 10:07:58.789767 14521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:58.803046 14521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:58.805707 14521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:58.805917 14521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:58.810768 14521 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:43205
I20260925 10:07:58.810806 14588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:43205 every 8 connection(s)
I20260925 10:07:58.811244 14521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:07:58.812230 14589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:07:58.814848 14589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc: Bootstrap starting.
I20260925 10:07:58.815447 14589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc: Neither blocks nor log segments found. Creating new log.
I20260925 10:07:58.815696 14589 log.cc:824] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc: Log is configured to *not* fsync() on all Append() calls
I20260925 10:07:58.816532 14589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc: No bootstrap required, opened a new log
I20260925 10:07:58.818203 14589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6911b7c9f2ba4a1690e9bc6f43f29dbc" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43205 } }
I20260925 10:07:58.818356 14589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:07:58.818379 14589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6911b7c9f2ba4a1690e9bc6f43f29dbc, State: Initialized, Role: FOLLOWER
I20260925 10:07:58.818552 14589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6911b7c9f2ba4a1690e9bc6f43f29dbc" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43205 } }
I20260925 10:07:58.818641 14589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:07:58.818665 14589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:07:58.818696 14589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:07:58.819237 14589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6911b7c9f2ba4a1690e9bc6f43f29dbc" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43205 } }
I20260925 10:07:58.819321 14589 leader_election.cc:302] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6911b7c9f2ba4a1690e9bc6f43f29dbc; no voters:
I20260925 10:07:58.819464 14589 leader_election.cc:288] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:07:58.819530 14592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:07:58.819720 14592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc [term 1 LEADER]: Becoming Leader. State: Replica: 6911b7c9f2ba4a1690e9bc6f43f29dbc, State: Running, Role: LEADER
I20260925 10:07:58.819802 14589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:07:58.819809 14592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6911b7c9f2ba4a1690e9bc6f43f29dbc" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43205 } }
I20260925 10:07:58.820325 14594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6911b7c9f2ba4a1690e9bc6f43f29dbc. Latest consensus state: current_term: 1 leader_uuid: "6911b7c9f2ba4a1690e9bc6f43f29dbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6911b7c9f2ba4a1690e9bc6f43f29dbc" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43205 } } }
I20260925 10:07:58.820410 14594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:58.820576 14593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6911b7c9f2ba4a1690e9bc6f43f29dbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6911b7c9f2ba4a1690e9bc6f43f29dbc" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43205 } } }
I20260925 10:07:58.820642 14593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc [sys.catalog]: This master's current role is: LEADER
I20260925 10:07:58.820791 14600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:07:58.820833 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 14521
I20260925 10:07:58.820971 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:07:58.821373 14600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:07:58.821816 14501 external_mini_cluster.cc:1428] Setting key 17114c3f9748464657d34032c6ab7ca3
I20260925 10:07:58.823395 14600 catalog_manager.cc:1347] Generated new cluster ID: 73b29bbad54f4d8fb2dda95a8cccc765
I20260925 10:07:58.823470 14600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:07:58 dist-test-slave-s6tm krb5kdc[14508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:07:58.842232 14600 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:07:58.842363 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:58.822751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.14.41.126:43205)
0925 10:07:58.822967 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:58.822979 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:58.823381 (+ 402us) server_negotiation.cc:407] Connection header received
0925 10:07:58.825627 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:58.825635 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:58.825706 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:58.825870 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:58.828049 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:58.830054 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:58.831596 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:58.832312 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:58.832449 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:58.837300 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:58.837307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:58.837307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:58.837322 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:58.839537 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:58.840230 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:58.840234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:58.840235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:58.840383 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:58.840750 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:58.840751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:58.840751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:58.841095 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:58.841267 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:58.841808 (+ 541us) server_negotiation.cc:299] Negotiation successful
0925 10:07:58.842011 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260925 10:07:58.842928 14600 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:07:58.847847 14600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6911b7c9f2ba4a1690e9bc6f43f29dbc: Generated new TSK 0
I20260925 10:07:58.848196 14600 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
I20260925 10:07:58.880563 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:43205
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:41933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:58.968200 14615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:58.968482 14615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:58.968529 14615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:58.969813 14615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:58.969911 14615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:58.969964 14615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:07:58.971582 14615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:41933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:43205
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:58.971901 14615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:58.972188 14615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:58.978689 14620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:58.978791 14623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:58.978842 14621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:58.979028 14615 server_base.cc:1034] running on GCE node
I20260925 10:07:58.979286 14615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:58.979578 14615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:58.980794 14615 hybrid_clock.cc:648] HybridClock initialized: now 1790330878980765 us; error 42 us; skew 500 ppm
Sep 25 10:07:58 dist-test-slave-s6tm krb5kdc[14508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330878, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:58.983919 14615 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:07:58.984513 14615 webserver.cc:460] Webserver started at http://127.14.41.65:41947/ using document root <none> and password file <none>
I20260925 10:07:58.984800 14615 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:58.984884 14615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:58.985044 14615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:58.986259 14615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "67fad7209e03416b911d4665f952dc2d"
format_stamp: "Formatted at 2026-09-25 10:07:58 on dist-test-slave-s6tm"
server_key: "0553926f30732deaf6b46e36740fbab9"
server_key_iv: "6cdd68b13fd8e64e23b97edfbe0c0af9"
server_key_version: "encryptionkey@0"
I20260925 10:07:58.986671 14615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "67fad7209e03416b911d4665f952dc2d"
format_stamp: "Formatted at 2026-09-25 10:07:58 on dist-test-slave-s6tm"
server_key: "0553926f30732deaf6b46e36740fbab9"
server_key_iv: "6cdd68b13fd8e64e23b97edfbe0c0af9"
server_key_version: "encryptionkey@0"
I20260925 10:07:58.988284 14615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:07:58.989218 14630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:58.989396 14615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:07:58.989557 14615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "67fad7209e03416b911d4665f952dc2d"
format_stamp: "Formatted at 2026-09-25 10:07:58 on dist-test-slave-s6tm"
server_key: "0553926f30732deaf6b46e36740fbab9"
server_key_iv: "6cdd68b13fd8e64e23b97edfbe0c0af9"
server_key_version: "encryptionkey@0"
I20260925 10:07:58.989641 14615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:59.017241 14615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:59.020242 14615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:59.020463 14615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:59.020963 14615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:07:59.021379 14615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:07:59.021420 14615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:59.021443 14615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:07:59.021456 14615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:59.028288 14615 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:34851
I20260925 10:07:59.028491 14743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:34851 every 8 connection(s)
I20260925 10:07:59.028740 14615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
Sep 25 10:07:59 dist-test-slave-s6tm krb5kdc[14508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330878, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:07:59.036952 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 14615
I20260925 10:07:59.037096 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:07:59.037266 14501 external_mini_cluster.cc:1428] Setting key 2f79b8451a5907c0dc9e441c5e259093
I20260925 10:07:59.040740 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:59.029578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:56323 (local address 127.14.41.126:43205)
0925 10:07:59.029628 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:59.029630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:59.029857 (+ 227us) server_negotiation.cc:407] Connection header received
0925 10:07:59.030385 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:59.030386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:59.030397 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:59.030461 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:59.031690 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:59.032448 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:59.033693 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:59.033922 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:59.033998 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:59.036695 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:59.036697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:59.036697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:59.036734 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:59.038826 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:59.039459 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:59.039461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:59.039461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:59.039538 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:59.039865 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:59.039866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:59.039866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:59.040075 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:59.040157 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:59.040506 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 10:07:59.040630 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:07:59.040963 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:59.029537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43205 (local address 127.14.41.65:56323)
0925 10:07:59.029707 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:59.029751 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:59.030264 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:59.030541 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:59.030544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:59.031359 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:59.031572 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:59.031578 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:59.032556 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:59.032557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:59.033578 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:59.033581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:59.034031 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:59.034032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:59.034158 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:59.034725 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:59.034734 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:59.036560 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:59.039026 (+ 2466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:59.039049 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:59.039052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:59.039363 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:59.039621 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:59.039622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:59.039623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:59.039770 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:59.040160 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:59.040169 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:59.040345 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 10:07:59.040481 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 10:07:59.040678 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260925 10:07:59.041796 14744 heartbeater.cc:347] Connected to a master server at 127.14.41.126:43205
I20260925 10:07:59.041925 14744 heartbeater.cc:464] Registering TS with master...
I20260925 10:07:59.042166 14744 heartbeater.cc:511] Master 127.14.41.126:43205 requested a full tablet report, sending...
I20260925 10:07:59.042917 14553 ts_manager.cc:194] Registered new tserver with Master: 67fad7209e03416b911d4665f952dc2d (127.14.41.65:34851)
I20260925 10:07:59.044054 14553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:56323
WARNING: no policy specified for kudu/127.14.41.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
I20260925 10:07:59.076664 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:43205
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:41933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:59.162469 14749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:59.162709 14749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:59.162736 14749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:59.164003 14749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:59.164052 14749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:59.164083 14749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:07:59.165700 14749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:41933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:43205
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:59.166000 14749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:59.166249 14749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:07:59.172291 14749 server_base.cc:1034] running on GCE node
W20260925 10:07:59.172256 14755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:59.172250 14754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:59.172248 14757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:59.172668 14749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:59.172921 14749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:59.174068 14749 hybrid_clock.cc:648] HybridClock initialized: now 1790330879174029 us; error 60 us; skew 500 ppm
Sep 25 10:07:59 dist-test-slave-s6tm krb5kdc[14508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330879, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:59.176862 14749 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:07:59.177361 14749 webserver.cc:460] Webserver started at http://127.14.41.66:41373/ using document root <none> and password file <none>
I20260925 10:07:59.177629 14749 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:59.177701 14749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:59.177855 14749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:59.179033 14749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "8095f2deb7f04ad2b5519f36e09bbe1e"
format_stamp: "Formatted at 2026-09-25 10:07:59 on dist-test-slave-s6tm"
server_key: "2706dfd22e0f6f86fd9be9e43cb7f2da"
server_key_iv: "c4c07d14fe4f4cb026b95b30c223149e"
server_key_version: "encryptionkey@0"
I20260925 10:07:59.179423 14749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "8095f2deb7f04ad2b5519f36e09bbe1e"
format_stamp: "Formatted at 2026-09-25 10:07:59 on dist-test-slave-s6tm"
server_key: "2706dfd22e0f6f86fd9be9e43cb7f2da"
server_key_iv: "c4c07d14fe4f4cb026b95b30c223149e"
server_key_version: "encryptionkey@0"
I20260925 10:07:59.180915 14749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:07:59.181737 14764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:59.181962 14749 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:07:59.182080 14749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "8095f2deb7f04ad2b5519f36e09bbe1e"
format_stamp: "Formatted at 2026-09-25 10:07:59 on dist-test-slave-s6tm"
server_key: "2706dfd22e0f6f86fd9be9e43cb7f2da"
server_key_iv: "c4c07d14fe4f4cb026b95b30c223149e"
server_key_version: "encryptionkey@0"
I20260925 10:07:59.182163 14749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:59.191134 14749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:59.194227 14749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:59.194408 14749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:59.194679 14749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:07:59.195027 14749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:07:59.195086 14749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:59.195120 14749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:07:59.195173 14749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:59.201809 14749 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:38275
I20260925 10:07:59.201853 14877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:38275 every 8 connection(s)
I20260925 10:07:59.202247 14749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:07:59.202685 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 14749
I20260925 10:07:59.202800 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:07:59.202972 14501 external_mini_cluster.cc:1428] Setting key 0d2cf5f8042545acd7b1c3ce169dd8f0
Sep 25 10:07:59 dist-test-slave-s6tm krb5kdc[14508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330879, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.41.67@KRBTEST.COM; defaulting to no policy
I20260925 10:07:59.215170 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:59.203129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:56151 (local address 127.14.41.126:43205)
0925 10:07:59.203191 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:59.203193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:59.203431 (+ 238us) server_negotiation.cc:407] Connection header received
0925 10:07:59.204196 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:59.204198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:59.204213 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:59.204290 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:59.205549 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:59.206088 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:59.207706 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:59.207976 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:59.208068 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:59.210859 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:59.210861 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:59.210861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:59.210871 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:59.212952 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:59.213672 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:59.213674 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:59.213674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:59.213829 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:59.214256 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:59.214257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:59.214257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:59.214481 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:59.214558 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:59.214995 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 10:07:59.215091 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 10:07:59.215529 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:59.203085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43205 (local address 127.14.41.66:56151)
0925 10:07:59.203288 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:59.203342 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:59.204087 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:59.204348 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:59.204351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:59.205050 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:59.205386 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:59.205394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:59.206176 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:59.206178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:59.207571 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:59.207575 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:59.208083 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:59.208084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:59.208293 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:59.208912 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:59.208926 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:59.210732 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:59.213217 (+ 2485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:59.213227 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:59.213230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:59.213549 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:59.213977 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:59.213978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:59.213979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:59.214156 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:59.214579 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:59.214586 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:59.214742 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 10:07:59.214947 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 10:07:59.215113 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.67@KRBTEST.COM" created.
I20260925 10:07:59.216471 14878 heartbeater.cc:347] Connected to a master server at 127.14.41.126:43205
I20260925 10:07:59.216600 14878 heartbeater.cc:464] Registering TS with master...
I20260925 10:07:59.216838 14878 heartbeater.cc:511] Master 127.14.41.126:43205 requested a full tablet report, sending...
I20260925 10:07:59.217473 14553 ts_manager.cc:194] Registered new tserver with Master: 8095f2deb7f04ad2b5519f36e09bbe1e (127.14.41.66:38275)
I20260925 10:07:59.217900 14553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:56151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
I20260925 10:07:59.236204 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:43205
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:41933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:07:59.321226 14883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:07:59.321523 14883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:07:59.321553 14883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:07:59.322791 14883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:07:59.322840 14883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:07:59.322870 14883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:07:59.324469 14883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:41933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:43205
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:07:59.324733 14883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:07:59.324985 14883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:07:59.330654 14889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:59.330893 14883 server_base.cc:1034] running on GCE node
W20260925 10:07:59.330751 14888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:07:59.330755 14891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:07:59.331257 14883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:07:59.331576 14883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:07:59.332741 14883 hybrid_clock.cc:648] HybridClock initialized: now 1790330879332738 us; error 39 us; skew 500 ppm
Sep 25 10:07:59 dist-test-slave-s6tm krb5kdc[14508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330879, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:07:59.335513 14883 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:07:59.336117 14883 webserver.cc:460] Webserver started at http://127.14.41.67:41347/ using document root <none> and password file <none>
I20260925 10:07:59.336436 14883 fs_manager.cc:359] Metadata directory not provided
I20260925 10:07:59.336516 14883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:07:59.336679 14883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:07:59.337779 14883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "2f0d8e3a6fa248049a6456fc7eedacd1"
format_stamp: "Formatted at 2026-09-25 10:07:59 on dist-test-slave-s6tm"
server_key: "83064f5028350615474258d754a9a847"
server_key_iv: "ee9c0a85a8da24f7ebe7d3650ceb0a8c"
server_key_version: "encryptionkey@0"
I20260925 10:07:59.338151 14883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "2f0d8e3a6fa248049a6456fc7eedacd1"
format_stamp: "Formatted at 2026-09-25 10:07:59 on dist-test-slave-s6tm"
server_key: "83064f5028350615474258d754a9a847"
server_key_iv: "ee9c0a85a8da24f7ebe7d3650ceb0a8c"
server_key_version: "encryptionkey@0"
I20260925 10:07:59.339610 14883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:07:59.340552 14898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:59.340876 14883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:07:59.341044 14883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "2f0d8e3a6fa248049a6456fc7eedacd1"
format_stamp: "Formatted at 2026-09-25 10:07:59 on dist-test-slave-s6tm"
server_key: "83064f5028350615474258d754a9a847"
server_key_iv: "ee9c0a85a8da24f7ebe7d3650ceb0a8c"
server_key_version: "encryptionkey@0"
I20260925 10:07:59.341125 14883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:07:59.361264 14883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:07:59.363694 14883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:07:59.363889 14883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:07:59.364207 14883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:07:59.364569 14883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:07:59.364635 14883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:59.364678 14883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:07:59.364705 14883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:07:59.370975 14883 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:38735
I20260925 10:07:59.371016 15011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:38735 every 8 connection(s)
I20260925 10:07:59.371348 14883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:07:59.371915 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 14883
I20260925 10:07:59.372020 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:07:59.372268 14501 external_mini_cluster.cc:1428] Setting key a92c657a021f2c3f6d6872fd7e83826d
Sep 25 10:07:59 dist-test-slave-s6tm krb5kdc[14508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330879, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:07:59.382017 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:59.372238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:43637 (local address 127.14.41.126:43205)
0925 10:07:59.372292 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:59.372294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:59.372488 (+ 194us) server_negotiation.cc:407] Connection header received
0925 10:07:59.373242 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:59.373243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:59.373255 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:59.373303 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:59.374425 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:59.375146 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:59.376390 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:59.376588 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:59.376653 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:59.379186 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:59.379188 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:59.379188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:59.379195 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:59.380677 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:59.381145 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:59.381146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:59.381146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:59.381201 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:59.381445 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:59.381445 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:59.381446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:59.381558 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:59.381610 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:59.381873 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 10:07:59.381946 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:07:59.382296 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:59.372133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43205 (local address 127.14.41.67:43637)
0925 10:07:59.372336 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 10:07:59.372396 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 10:07:59.373141 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:07:59.373371 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:07:59.373374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:07:59.374089 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:07:59.374309 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:59.374314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:59.375254 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:59.375256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:59.376300 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:07:59.376304 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:07:59.376696 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:07:59.376697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:07:59.376825 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:59.377492 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:07:59.377501 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:07:59.379085 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:07:59.380765 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:59.380774 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:59.380776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:59.381066 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:59.381266 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:07:59.381267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:07:59.381267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:07:59.381381 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:07:59.381616 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:07:59.381620 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:07:59.381755 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 10:07:59.381846 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 10:07:59.381985 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":89,"threads_started":1}
I20260925 10:07:59.383016 15012 heartbeater.cc:347] Connected to a master server at 127.14.41.126:43205
I20260925 10:07:59.383111 15012 heartbeater.cc:464] Registering TS with master...
I20260925 10:07:59.383271 15012 heartbeater.cc:511] Master 127.14.41.126:43205 requested a full tablet report, sending...
I20260925 10:07:59.383674 14553 ts_manager.cc:194] Registered new tserver with Master: 2f0d8e3a6fa248049a6456fc7eedacd1 (127.14.41.67:38735)
I20260925 10:07:59.384153 14553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:43637
I20260925 10:07:59.384649 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:07:59 dist-test-slave-s6tm krb5kdc[14508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330879, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:07:59 dist-test-slave-s6tm krb5kdc[14508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330879, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.41.65@KRBTEST.COM
I20260925 10:07:59.410205 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:59.401073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.14.41.65:34851)
0925 10:07:59.401308 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:59.401315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:59.401411 (+ 96us) server_negotiation.cc:407] Connection header received
0925 10:07:59.401500 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:59.401503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:59.401520 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:59.401607 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:59.402225 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:59.403123 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:59.403888 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:59.404295 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:59.404395 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:59.407040 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:59.407049 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:59.407050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:59.407059 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:59.408916 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:59.409317 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:59.409320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:59.409321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:59.409381 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:59.409591 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:59.409592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:59.409592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:59.409739 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:59.409931 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:59.409977 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 10:07:59.410067 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":84,"threads_started":1}
W20260925 10:07:59.410742 14658 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33194
Sep 25 10:07:59 dist-test-slave-s6tm krb5kdc[14508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:07:59 dist-test-slave-s6tm krb5kdc[14508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.65@KRBTEST.COM
I20260925 10:07:59.434029 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:07:59.426575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.14.41.65:34851)
0925 10:07:59.426662 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 10:07:59.426664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:07:59.426673 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:07:59.426701 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:07:59.426702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:07:59.426713 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:07:59.426783 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:07:59.427460 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:59.428089 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:59.428816 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:07:59.429036 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:07:59.429092 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:07:59.431487 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:07:59.431488 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:07:59.431488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:07:59.431496 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:07:59.432995 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:59.433334 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:59.433335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:59.433335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:59.433390 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:07:59.433611 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:07:59.433612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:07:59.433612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:07:59.433715 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:07:59.433811 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:07:59.433883 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 10:07:59.433954 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 10:07:59.434762 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 14615
I20260925 10:07:59.440256 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 14749
I20260925 10:07:59.446000 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 14883
I20260925 10:07:59.450421 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 14521
2026-09-25T10:07:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1977 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:07:59 dist-test-slave-s6tm krb5kdc[15026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:07:59 dist-test-slave-s6tm krb5kdc[15026](info): set up 2 sockets
Sep 25 10:07:59 dist-test-slave-s6tm krb5kdc[15026](info): commencing operation
krb5kdc: starting...
W20260925 10:08:00.516987 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:08:00 dist-test-slave-s6tm krb5kdc[15026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:08:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:08:00.642944 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:40483
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:40483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:00.724809 15039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:00.725098 15039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:00.725140 15039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:00.726372 15039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:00.726454 15039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:00.726477 15039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:00.726498 15039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:00.726529 15039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:08:00.728121 15039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:40483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:40483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:00.728428 15039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:00.728694 15039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:00.733938 15047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:00.733961 15045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:00.733946 15044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:00.734218 15039 server_base.cc:1034] running on GCE node
I20260925 10:08:00.734480 15039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:00.734740 15039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:00.735911 15039 hybrid_clock.cc:648] HybridClock initialized: now 1790330880735894 us; error 34 us; skew 500 ppm
Sep 25 10:08:00 dist-test-slave-s6tm krb5kdc[15026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330880, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:00.745450 15039 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:08:00.745970 15039 webserver.cc:460] Webserver started at http://127.14.41.126:36691/ using document root <none> and password file <none>
I20260925 10:08:00.746181 15039 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:00.746227 15039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:00.746325 15039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:00.747408 15039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "78c9244b3f0d405186d0546ce0563419"
format_stamp: "Formatted at 2026-09-25 10:08:00 on dist-test-slave-s6tm"
server_key: "8ec03ab1d1a8e2ded4bcfdf3f05685a8"
server_key_iv: "0c051d130f4cedee82eb368e75e2dc4a"
server_key_version: "encryptionkey@0"
I20260925 10:08:00.747738 15039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "78c9244b3f0d405186d0546ce0563419"
format_stamp: "Formatted at 2026-09-25 10:08:00 on dist-test-slave-s6tm"
server_key: "8ec03ab1d1a8e2ded4bcfdf3f05685a8"
server_key_iv: "0c051d130f4cedee82eb368e75e2dc4a"
server_key_version: "encryptionkey@0"
I20260925 10:08:00.749243 15039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:08:00.750131 15054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:00.750432 15039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:00.750545 15039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "78c9244b3f0d405186d0546ce0563419"
format_stamp: "Formatted at 2026-09-25 10:08:00 on dist-test-slave-s6tm"
server_key: "8ec03ab1d1a8e2ded4bcfdf3f05685a8"
server_key_iv: "0c051d130f4cedee82eb368e75e2dc4a"
server_key_version: "encryptionkey@0"
I20260925 10:08:00.750627 15039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:00.763108 15039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:00.766058 15039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:00.766234 15039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:00.770871 15039 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:40483
I20260925 10:08:00.770925 15106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:40483 every 8 connection(s)
I20260925 10:08:00.771346 15039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:00.772336 15107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:00.774780 15107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419: Bootstrap starting.
I20260925 10:08:00.775424 15107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:00.775743 15107 log.cc:824] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:00.776485 15107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419: No bootstrap required, opened a new log
I20260925 10:08:00.777959 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 15039
I20260925 10:08:00.778035 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:08:00.778213 14501 external_mini_cluster.cc:1428] Setting key a4ea109bfb82c8f4fe96d7d9da7caf82
I20260925 10:08:00.778321 15107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c9244b3f0d405186d0546ce0563419" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 40483 } }
I20260925 10:08:00.778481 15107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:00.778529 15107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78c9244b3f0d405186d0546ce0563419, State: Initialized, Role: FOLLOWER
I20260925 10:08:00.778679 15107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c9244b3f0d405186d0546ce0563419" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 40483 } }
I20260925 10:08:00.778757 15107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:08:00.778790 15107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:00.778848 15107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:00.780153 15107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c9244b3f0d405186d0546ce0563419" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 40483 } }
I20260925 10:08:00.780285 15107 leader_election.cc:302] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78c9244b3f0d405186d0546ce0563419; no voters:
I20260925 10:08:00.780454 15107 leader_election.cc:288] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:08:00.780514 15112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:00.780679 15112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419 [term 1 LEADER]: Becoming Leader. State: Replica: 78c9244b3f0d405186d0546ce0563419, State: Running, Role: LEADER
I20260925 10:08:00.780754 15107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:00.780781 15112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c9244b3f0d405186d0546ce0563419" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 40483 } }
I20260925 10:08:00.781325 15112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78c9244b3f0d405186d0546ce0563419. Latest consensus state: current_term: 1 leader_uuid: "78c9244b3f0d405186d0546ce0563419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c9244b3f0d405186d0546ce0563419" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 40483 } } }
I20260925 10:08:00.781420 15112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:00.781335 15113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78c9244b3f0d405186d0546ce0563419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78c9244b3f0d405186d0546ce0563419" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 40483 } } }
I20260925 10:08:00.781476 15113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419 [sys.catalog]: This master's current role is: LEADER
W20260925 10:08:00.782761 15127 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:08:00.782821 15127 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:08:00.782886 15121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:00.783061 15121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:00.784767 15121 catalog_manager.cc:1347] Generated new cluster ID: f7c133dcbb5a4ea9a1e4fc8bd74f2c48
I20260925 10:08:00.784822 15121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:08:00 dist-test-slave-s6tm krb5kdc[15026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:00.790961 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:00.778765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.14.41.126:40483)
0925 10:08:00.778964 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:00.778972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:00.779026 (+ 54us) server_negotiation.cc:407] Connection header received
0925 10:08:00.779176 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:00.779182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:00.779233 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:00.779343 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:00.780378 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:00.781905 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:00.783080 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:00.783539 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:00.783678 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:00.786753 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:00.786758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:00.786758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:00.786772 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:00.788966 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:00.789348 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:00.789351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:00.789352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:00.789484 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:00.789658 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:00.789659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:00.789659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:00.789961 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:00.790112 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:00.790590 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 10:08:00.790733 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":79,"threads_started":1}
I20260925 10:08:00.806439 15121 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:08:00.807303 15121 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:00.816252 15121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78c9244b3f0d405186d0546ce0563419: Generated new TSK 0
I20260925 10:08:00.816547 15121 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
I20260925 10:08:00.851397 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:40483
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:00.936955 15133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:00.937208 15133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:00.937249 15133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:00.938427 15133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:00.938499 15133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:00.938545 15133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:08:00.940217 15133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:40483
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:00.940512 15133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:00.940750 15133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:00.946039 15141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:00.946027 15139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:00.946027 15138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:00.946054 15133 server_base.cc:1034] running on GCE node
I20260925 10:08:00.946497 15133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:00.946774 15133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:00.947994 15133 hybrid_clock.cc:648] HybridClock initialized: now 1790330880947968 us; error 27 us; skew 500 ppm
Sep 25 10:08:00 dist-test-slave-s6tm krb5kdc[15026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330880, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:00.950996 15133 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:08:00.951506 15133 webserver.cc:460] Webserver started at http://127.14.41.65:38481/ using document root <none> and password file <none>
I20260925 10:08:00.951783 15133 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:00.951839 15133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:00.952018 15133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:00.953173 15133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "f9f09b7978d648ffa146014037447381"
format_stamp: "Formatted at 2026-09-25 10:08:00 on dist-test-slave-s6tm"
server_key: "75eba0688f3bac2ebd65363dbc41e492"
server_key_iv: "e90a3e1a5c4db5642cc70ae4ed46be59"
server_key_version: "encryptionkey@0"
I20260925 10:08:00.953581 15133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "f9f09b7978d648ffa146014037447381"
format_stamp: "Formatted at 2026-09-25 10:08:00 on dist-test-slave-s6tm"
server_key: "75eba0688f3bac2ebd65363dbc41e492"
server_key_iv: "e90a3e1a5c4db5642cc70ae4ed46be59"
server_key_version: "encryptionkey@0"
I20260925 10:08:00.955080 15133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:08:00.955919 15148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:00.956143 15133 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:00.956238 15133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "f9f09b7978d648ffa146014037447381"
format_stamp: "Formatted at 2026-09-25 10:08:00 on dist-test-slave-s6tm"
server_key: "75eba0688f3bac2ebd65363dbc41e492"
server_key_iv: "e90a3e1a5c4db5642cc70ae4ed46be59"
server_key_version: "encryptionkey@0"
I20260925 10:08:00.956311 15133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:00.966801 15133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:00.969645 15133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:00.969856 15133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:00.970144 15133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:00.970482 15133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:00.970539 15133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:00.970582 15133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:00.970608 15133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:00.977012 15133 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:33421
I20260925 10:08:00.977052 15261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:33421 every 8 connection(s)
I20260925 10:08:00.977417 15133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
Sep 25 10:08:00 dist-test-slave-s6tm krb5kdc[15026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330880, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:00.987340 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 15133
I20260925 10:08:00.987555 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:08:00.987747 14501 external_mini_cluster.cc:1428] Setting key 5fc18a42a5118604974f1c17966bceb8
I20260925 10:08:00.988703 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:00.978210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:37617 (local address 127.14.41.126:40483)
0925 10:08:00.978294 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:00.978296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:00.978487 (+ 191us) server_negotiation.cc:407] Connection header received
0925 10:08:00.979439 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:00.979441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:00.979460 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:00.979528 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:00.980284 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:00.980929 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:00.982152 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:00.982408 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:00.982473 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:00.985338 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:00.985340 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:00.985340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:00.985347 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:00.986794 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:00.987306 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:00.987307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:00.987308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:00.987374 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:00.987775 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:00.987776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:00.987776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:00.987980 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:00.988048 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:00.988471 (+ 423us) server_negotiation.cc:299] Negotiation successful
0925 10:08:00.988596 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:08:00.988890 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:00.978132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:40483 (local address 127.14.41.65:37617)
0925 10:08:00.978339 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:00.978391 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:00.979334 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:00.979614 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:00.979618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:00.979906 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:00.980192 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:00.980196 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:00.981072 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:00.981073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:00.982076 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:00.982078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:00.982723 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:00.982724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:00.982951 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:00.983614 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:00.983622 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:00.985235 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:00.986888 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:00.986895 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:00.986897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:00.987202 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:00.987511 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:00.987512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:00.987512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:00.987703 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:00.988060 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:00.988067 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:00.988258 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 10:08:00.988380 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 10:08:00.988598 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260925 10:08:00.989707 15262 heartbeater.cc:347] Connected to a master server at 127.14.41.126:40483
I20260925 10:08:00.989822 15262 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:00.990186 15262 heartbeater.cc:511] Master 127.14.41.126:40483 requested a full tablet report, sending...
I20260925 10:08:00.990764 15071 ts_manager.cc:194] Registered new tserver with Master: f9f09b7978d648ffa146014037447381 (127.14.41.65:33421)
I20260925 10:08:00.991427 15071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:37617
WARNING: no policy specified for kudu/127.14.41.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
I20260925 10:08:01.015149 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:40483
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:01.098711 15267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:01.098956 15267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:01.098982 15267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:01.100340 15267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:01.100387 15267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:01.100418 15267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:08:01.102069 15267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:40483
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:01.102349 15267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:01.102612 15267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:08:01.108093 15267 server_base.cc:1034] running on GCE node
W20260925 10:08:01.108119 15273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:01.108088 15272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:01.108121 15275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:01.108513 15267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:01.108776 15267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:01.110008 15267 hybrid_clock.cc:648] HybridClock initialized: now 1790330881109995 us; error 29 us; skew 500 ppm
Sep 25 10:08:01 dist-test-slave-s6tm krb5kdc[15026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330881, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:01.112636 15267 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:08:01.113052 15267 webserver.cc:460] Webserver started at http://127.14.41.66:43631/ using document root <none> and password file <none>
I20260925 10:08:01.113256 15267 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:01.113298 15267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:01.113390 15267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:01.114410 15267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "87571b48e90e4003b644007f6e0eb468"
format_stamp: "Formatted at 2026-09-25 10:08:01 on dist-test-slave-s6tm"
server_key: "3d98e5612dfe470d0eb25906184e0ae4"
server_key_iv: "712928eba94110c8b48bd2895826b969"
server_key_version: "encryptionkey@0"
I20260925 10:08:01.114705 15267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "87571b48e90e4003b644007f6e0eb468"
format_stamp: "Formatted at 2026-09-25 10:08:01 on dist-test-slave-s6tm"
server_key: "3d98e5612dfe470d0eb25906184e0ae4"
server_key_iv: "712928eba94110c8b48bd2895826b969"
server_key_version: "encryptionkey@0"
I20260925 10:08:01.116065 15267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 10:08:01.116775 15282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:01.116995 15267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:01.117067 15267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "87571b48e90e4003b644007f6e0eb468"
format_stamp: "Formatted at 2026-09-25 10:08:01 on dist-test-slave-s6tm"
server_key: "3d98e5612dfe470d0eb25906184e0ae4"
server_key_iv: "712928eba94110c8b48bd2895826b969"
server_key_version: "encryptionkey@0"
I20260925 10:08:01.117123 15267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:01.142616 15267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:01.145102 15267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:01.145223 15267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:01.145406 15267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:01.145746 15267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:01.145782 15267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:01.145828 15267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:01.145854 15267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:01.152148 15267 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:36997
I20260925 10:08:01.152273 15397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:36997 every 8 connection(s)
I20260925 10:08:01.152578 15267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:08:01 dist-test-slave-s6tm krb5kdc[15026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330881, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:01.161020 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 15267
I20260925 10:08:01.161103 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:08:01.161319 14501 external_mini_cluster.cc:1428] Setting key 17b2cf4b07d46d272498732c326420ce
I20260925 10:08:01.164304 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:01.153523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:57537 (local address 127.14.41.126:40483)
0925 10:08:01.153571 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:01.153573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:01.153791 (+ 218us) server_negotiation.cc:407] Connection header received
0925 10:08:01.154788 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:01.154790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:01.154801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:01.154860 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:01.155493 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.156076 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.157081 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.157275 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.157331 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:01.160033 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:01.160035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:01.160035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:01.160043 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:01.162076 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:01.162710 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:01.162712 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:01.162712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:01.162780 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:01.163284 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:01.163286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:01.163286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:01.163404 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:01.163474 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:01.164036 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 10:08:01.164156 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:08:01.164322 15400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:01.153512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:40483 (local address 127.14.41.66:57537)
0925 10:08:01.153667 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:01.153702 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:01.154684 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:01.154929 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:01.154932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:01.155197 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:01.155394 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:01.155400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.156154 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.156155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:01.157000 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:01.157002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.157370 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.157371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:01.157497 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:01.158137 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:01.158152 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:01.159801 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:01.162164 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:01.162174 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:01.162178 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:01.162620 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:01.162961 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:01.162962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:01.162962 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:01.163188 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:01.163490 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:01.163498 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:01.163707 (+ 209us) client_negotiation.cc:769] Sending connection context
0925 10:08:01.163842 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 10:08:01.164106 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260925 10:08:01.164958 15398 heartbeater.cc:347] Connected to a master server at 127.14.41.126:40483
I20260925 10:08:01.165138 15398 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:01.165369 15398 heartbeater.cc:511] Master 127.14.41.126:40483 requested a full tablet report, sending...
I20260925 10:08:01.165831 15071 ts_manager.cc:194] Registered new tserver with Master: 87571b48e90e4003b644007f6e0eb468 (127.14.41.66:36997)
I20260925 10:08:01.166250 15071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:57537
WARNING: no policy specified for kudu/127.14.41.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
I20260925 10:08:01.192973 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:40483
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:01.277086 15403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:01.277304 15403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:01.277331 15403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:01.278570 15403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:01.278620 15403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:01.278648 15403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:08:01.280757 15403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:40483
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:01.281031 15403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:01.281299 15403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:01.286619 15411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:01.286619 15408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:01.286689 15409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:01.287144 15403 server_base.cc:1034] running on GCE node
I20260925 10:08:01.287379 15403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:01.287611 15403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:01.288766 15403 hybrid_clock.cc:648] HybridClock initialized: now 1790330881288752 us; error 27 us; skew 500 ppm
Sep 25 10:08:01 dist-test-slave-s6tm krb5kdc[15026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330881, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:01.291525 15403 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:08:01.292181 15403 webserver.cc:460] Webserver started at http://127.14.41.67:35867/ using document root <none> and password file <none>
I20260925 10:08:01.292445 15403 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:01.292493 15403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:01.292629 15403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:01.293736 15403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "6199c95b40964270b72e7276d5e25a6d"
format_stamp: "Formatted at 2026-09-25 10:08:01 on dist-test-slave-s6tm"
server_key: "457ebc1c2bd6639c37db782185e0a8c5"
server_key_iv: "c4598e1e8c11b15650632f3a778fa340"
server_key_version: "encryptionkey@0"
I20260925 10:08:01.294206 15403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "6199c95b40964270b72e7276d5e25a6d"
format_stamp: "Formatted at 2026-09-25 10:08:01 on dist-test-slave-s6tm"
server_key: "457ebc1c2bd6639c37db782185e0a8c5"
server_key_iv: "c4598e1e8c11b15650632f3a778fa340"
server_key_version: "encryptionkey@0"
I20260925 10:08:01.295599 15403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:08:01.296546 15418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:01.296757 15403 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:08:01.296859 15403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "6199c95b40964270b72e7276d5e25a6d"
format_stamp: "Formatted at 2026-09-25 10:08:01 on dist-test-slave-s6tm"
server_key: "457ebc1c2bd6639c37db782185e0a8c5"
server_key_iv: "c4598e1e8c11b15650632f3a778fa340"
server_key_version: "encryptionkey@0"
I20260925 10:08:01.296999 15403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:01.310480 15403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:01.312942 15403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:01.313143 15403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:01.313426 15403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:01.313752 15403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:01.313807 15403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:01.313843 15403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:01.313868 15403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:01.320129 15403 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:38615
I20260925 10:08:01.320204 15532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:38615 every 8 connection(s)
I20260925 10:08:01.320546 15403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
Sep 25 10:08:01 dist-test-slave-s6tm krb5kdc[15026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330881, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:01.328120 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 15403
I20260925 10:08:01.328222 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:08:01.328397 14501 external_mini_cluster.cc:1428] Setting key 6f54963601fc49b61df1520bafca82ef
I20260925 10:08:01.330991 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:01.321473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:58963 (local address 127.14.41.126:40483)
0925 10:08:01.321557 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:01.321559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:01.321732 (+ 173us) server_negotiation.cc:407] Connection header received
0925 10:08:01.322688 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:01.322690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:01.322701 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:01.322756 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:01.323429 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.323983 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.325062 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.325250 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.325304 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:01.327625 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:01.327627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:01.327627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:01.327634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:01.329699 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:01.330100 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:01.330100 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:01.330101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:01.330153 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:01.330381 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:01.330382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:01.330382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:01.330512 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:01.330562 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:01.330855 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 10:08:01.330925 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:08:01.331241 15535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:01.321410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:40483 (local address 127.14.41.67:58963)
0925 10:08:01.321593 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:01.321633 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:01.322585 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:01.322814 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:01.322818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:01.323083 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:01.323301 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:01.323305 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.324075 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.324077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:01.324981 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:01.324984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.325357 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.325358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:01.325477 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:01.326069 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:01.326079 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:01.327529 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:01.329785 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:01.329793 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:01.329794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:01.330028 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:01.330213 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:01.330214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:01.330214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:01.330323 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:01.330573 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:01.330579 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:01.330734 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 10:08:01.330833 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 10:08:01.330992 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":99,"threads_started":1}
I20260925 10:08:01.331727 15533 heartbeater.cc:347] Connected to a master server at 127.14.41.126:40483
I20260925 10:08:01.331815 15533 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:01.332108 15533 heartbeater.cc:511] Master 127.14.41.126:40483 requested a full tablet report, sending...
I20260925 10:08:01.332592 15071 ts_manager.cc:194] Registered new tserver with Master: 6199c95b40964270b72e7276d5e25a6d (127.14.41.67:38615)
I20260925 10:08:01.333097 15071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:58963
I20260925 10:08:01.340816 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:08:01.349223 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:01.342815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.14.41.126:40483)
0925 10:08:01.342885 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:01.342888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:01.342899 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:08:01.343000 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:01.343001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:01.343012 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:01.343062 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:01.343687 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.344277 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.345026 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.345221 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.345309 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:01.346533 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:01.346535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:01.346535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:01.346543 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:01.348023 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:01.348370 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:01.348371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:01.348371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:01.348431 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:01.348647 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:01.348647 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:01.348648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:01.348803 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:01.348930 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:01.349027 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 10:08:01.349111 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:08:01.351672 15071 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:08:01.353466 15071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:08:01.360610 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:01.356148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36966 (local address 127.14.41.67:38615)
0925 10:08:01.356623 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:01.356629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:01.356640 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:08:01.356751 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:01.356755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:01.356773 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:01.357066 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:01.357146 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.357795 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.359401 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.360154 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.360256 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:01.360305 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:01.360469 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 10:08:01.360515 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":103,"threads_started":1}
I20260925 10:08:01.360600 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:01.356047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:38615 (local address 127.0.0.1:36966)
0925 10:08:01.356432 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:01.356444 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:01.356500 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:01.356892 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:01.356895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:01.356897 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:01.356986 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:01.356990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.357928 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.357929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:01.359224 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:01.359226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.360259 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.360260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:01.360341 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:01.360357 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 10:08:01.360436 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 10:08:01.360495 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":42,"thread_start_us":73,"threads_started":1}
I20260925 10:08:01.360791 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:01.356264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53354 (local address 127.14.41.65:33421)
0925 10:08:01.356668 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:01.356672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:01.356681 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:08:01.356719 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:01.356721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:01.356737 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:01.356817 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:01.357058 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.357770 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.359300 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.360153 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.360228 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:01.360292 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:01.360644 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 10:08:01.360689 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":79,"threads_started":1}
I20260925 10:08:01.360605 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:01.356047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:33421 (local address 127.0.0.1:53354)
0925 10:08:01.356359 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:01.356403 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:01.356478 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:01.356915 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:01.356915 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:01.356917 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:01.356986 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:01.356991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.357876 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.357878 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:01.359208 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:01.359209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.360327 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.360328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:01.360399 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:01.360403 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:08:01.360480 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 10:08:01.360521 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":77,"thread_start_us":70,"threads_started":1}
I20260925 10:08:01.361404 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:01.356258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:36997 (local address 127.0.0.1:54412)
0925 10:08:01.356609 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:01.356617 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:01.356661 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:01.356911 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:01.356912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:01.356915 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:01.356986 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:01.356988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.359147 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.359147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:01.360109 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:01.360111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.361145 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.361146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:01.361220 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:01.361226 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:08:01.361273 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 10:08:01.361330 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":4,"thread_start_us":112,"threads_started":1}
I20260925 10:08:01.361658 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:01.356400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54412 (local address 127.14.41.66:36997)
0925 10:08:01.356586 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:01.356591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:01.356653 (+ 62us) server_negotiation.cc:407] Connection header received
0925 10:08:01.356725 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:01.356727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:01.356743 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:01.356817 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:01.357874 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.359041 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.360198 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.360982 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.361049 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:01.361109 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:01.361532 (+ 423us) server_negotiation.cc:299] Negotiation successful
0925 10:08:01.361582 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":86,"threads_started":1}
I20260925 10:08:01.361671 15196 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1b680ffa0a4dac92e144c2a180bc3f (DEFAULT_TABLE table=test-table [id=c30ebfb763e14b00921f844de2f14148]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:01.362082 15196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1b680ffa0a4dac92e144c2a180bc3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:01.362329 15331 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1b680ffa0a4dac92e144c2a180bc3f (DEFAULT_TABLE table=test-table [id=c30ebfb763e14b00921f844de2f14148]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:01.362643 15331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1b680ffa0a4dac92e144c2a180bc3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:01.365003 15550 tablet_bootstrap.cc:492] T 4b1b680ffa0a4dac92e144c2a180bc3f P f9f09b7978d648ffa146014037447381: Bootstrap starting.
I20260925 10:08:01.365384 15551 tablet_bootstrap.cc:492] T 4b1b680ffa0a4dac92e144c2a180bc3f P 87571b48e90e4003b644007f6e0eb468: Bootstrap starting.
I20260925 10:08:01.365682 15466 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1b680ffa0a4dac92e144c2a180bc3f (DEFAULT_TABLE table=test-table [id=c30ebfb763e14b00921f844de2f14148]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:01.365840 15550 tablet_bootstrap.cc:654] T 4b1b680ffa0a4dac92e144c2a180bc3f P f9f09b7978d648ffa146014037447381: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:01.366043 15551 tablet_bootstrap.cc:654] T 4b1b680ffa0a4dac92e144c2a180bc3f P 87571b48e90e4003b644007f6e0eb468: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:01.366037 15466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1b680ffa0a4dac92e144c2a180bc3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:01.366439 15550 log.cc:824] T 4b1b680ffa0a4dac92e144c2a180bc3f P f9f09b7978d648ffa146014037447381: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:01.366569 15551 log.cc:824] T 4b1b680ffa0a4dac92e144c2a180bc3f P 87571b48e90e4003b644007f6e0eb468: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:01.367301 15551 tablet_bootstrap.cc:492] T 4b1b680ffa0a4dac92e144c2a180bc3f P 87571b48e90e4003b644007f6e0eb468: No bootstrap required, opened a new log
I20260925 10:08:01.367442 15551 ts_tablet_manager.cc:1397] T 4b1b680ffa0a4dac92e144c2a180bc3f P 87571b48e90e4003b644007f6e0eb468: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:08:01.368615 15550 tablet_bootstrap.cc:492] T 4b1b680ffa0a4dac92e144c2a180bc3f P f9f09b7978d648ffa146014037447381: No bootstrap required, opened a new log
I20260925 10:08:01.368692 15550 ts_tablet_manager.cc:1397] T 4b1b680ffa0a4dac92e144c2a180bc3f P f9f09b7978d648ffa146014037447381: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 10:08:01.369175 15551 raft_consensus.cc:348] T 4b1b680ffa0a4dac92e144c2a180bc3f P 87571b48e90e4003b644007f6e0eb468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6199c95b40964270b72e7276d5e25a6d" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 38615 } } peers { permanent_uuid: "87571b48e90e4003b644007f6e0eb468" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 36997 } } peers { permanent_uuid: "f9f09b7978d648ffa146014037447381" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 33421 } }
I20260925 10:08:01.369381 15551 raft_consensus.cc:374] T 4b1b680ffa0a4dac92e144c2a180bc3f P 87571b48e90e4003b644007f6e0eb468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:01.369429 15551 raft_consensus.cc:729] T 4b1b680ffa0a4dac92e144c2a180bc3f P 87571b48e90e4003b644007f6e0eb468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87571b48e90e4003b644007f6e0eb468, State: Initialized, Role: FOLLOWER
I20260925 10:08:01.369541 15551 consensus_queue.cc:260] T 4b1b680ffa0a4dac92e144c2a180bc3f P 87571b48e90e4003b644007f6e0eb468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6199c95b40964270b72e7276d5e25a6d" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 38615 } } peers { permanent_uuid: "87571b48e90e4003b644007f6e0eb468" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 36997 } } peers { permanent_uuid: "f9f09b7978d648ffa146014037447381" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 33421 } }
I20260925 10:08:01.369598 15554 tablet_bootstrap.cc:492] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d: Bootstrap starting.
I20260925 10:08:01.369829 15551 ts_tablet_manager.cc:1428] T 4b1b680ffa0a4dac92e144c2a180bc3f P 87571b48e90e4003b644007f6e0eb468: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:08:01.369984 15398 heartbeater.cc:503] Master 127.14.41.126:40483 was elected leader, sending a full tablet report...
I20260925 10:08:01.370270 15554 tablet_bootstrap.cc:654] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:01.370826 15554 log.cc:824] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:01.370884 15550 raft_consensus.cc:348] T 4b1b680ffa0a4dac92e144c2a180bc3f P f9f09b7978d648ffa146014037447381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6199c95b40964270b72e7276d5e25a6d" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 38615 } } peers { permanent_uuid: "87571b48e90e4003b644007f6e0eb468" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 36997 } } peers { permanent_uuid: "f9f09b7978d648ffa146014037447381" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 33421 } }
I20260925 10:08:01.371065 15550 raft_consensus.cc:374] T 4b1b680ffa0a4dac92e144c2a180bc3f P f9f09b7978d648ffa146014037447381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:01.371109 15550 raft_consensus.cc:729] T 4b1b680ffa0a4dac92e144c2a180bc3f P f9f09b7978d648ffa146014037447381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9f09b7978d648ffa146014037447381, State: Initialized, Role: FOLLOWER
I20260925 10:08:01.371227 15550 consensus_queue.cc:260] T 4b1b680ffa0a4dac92e144c2a180bc3f P f9f09b7978d648ffa146014037447381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6199c95b40964270b72e7276d5e25a6d" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 38615 } } peers { permanent_uuid: "87571b48e90e4003b644007f6e0eb468" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 36997 } } peers { permanent_uuid: "f9f09b7978d648ffa146014037447381" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 33421 } }
I20260925 10:08:01.371539 15550 ts_tablet_manager.cc:1428] T 4b1b680ffa0a4dac92e144c2a180bc3f P f9f09b7978d648ffa146014037447381: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:08:01.371577 15262 heartbeater.cc:503] Master 127.14.41.126:40483 was elected leader, sending a full tablet report...
I20260925 10:08:01.371623 15554 tablet_bootstrap.cc:492] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d: No bootstrap required, opened a new log
I20260925 10:08:01.371670 15554 ts_tablet_manager.cc:1397] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:08:01.373296 15554 raft_consensus.cc:348] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6199c95b40964270b72e7276d5e25a6d" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 38615 } } peers { permanent_uuid: "87571b48e90e4003b644007f6e0eb468" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 36997 } } peers { permanent_uuid: "f9f09b7978d648ffa146014037447381" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 33421 } }
I20260925 10:08:01.373442 15554 raft_consensus.cc:374] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:01.373466 15554 raft_consensus.cc:729] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6199c95b40964270b72e7276d5e25a6d, State: Initialized, Role: FOLLOWER
I20260925 10:08:01.373591 15554 consensus_queue.cc:260] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6199c95b40964270b72e7276d5e25a6d" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 38615 } } peers { permanent_uuid: "87571b48e90e4003b644007f6e0eb468" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 36997 } } peers { permanent_uuid: "f9f09b7978d648ffa146014037447381" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 33421 } }
I20260925 10:08:01.373883 15554 ts_tablet_manager.cc:1428] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:08:01.373888 15533 heartbeater.cc:503] Master 127.14.41.126:40483 was elected leader, sending a full tablet report...
W20260925 10:08:01.403296 15399 tablet.cc:2369] T 4b1b680ffa0a4dac92e144c2a180bc3f P 87571b48e90e4003b644007f6e0eb468: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:08:01.478201 15263 tablet.cc:2369] T 4b1b680ffa0a4dac92e144c2a180bc3f P f9f09b7978d648ffa146014037447381: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:08:01.571185 15534 tablet.cc:2369] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:08:01.650874 15558 raft_consensus.cc:482] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:01.651027 15558 raft_consensus.cc:504] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6199c95b40964270b72e7276d5e25a6d" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 38615 } } peers { permanent_uuid: "87571b48e90e4003b644007f6e0eb468" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 36997 } } peers { permanent_uuid: "f9f09b7978d648ffa146014037447381" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 33421 } }
I20260925 10:08:01.651563 15558 leader_election.cc:288] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87571b48e90e4003b644007f6e0eb468 (127.14.41.66:36997), f9f09b7978d648ffa146014037447381 (127.14.41.65:33421)
I20260925 10:08:01.655824 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:01.651965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:44453 (local address 127.14.41.66:36997)
0925 10:08:01.652008 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:01.652010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:01.652019 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:08:01.652052 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:01.652053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:01.652061 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:01.652155 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:01.652326 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.653133 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.654753 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.655327 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.655479 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:01.655488 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:01.655711 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 10:08:01.655751 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:08:01.655838 15535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:01.651791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:36997 (local address 127.14.41.67:44453)
0925 10:08:01.651843 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:01.651856 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:01.651902 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:01.652170 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:01.652171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:01.652172 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:01.652230 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:01.652231 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.653248 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.653249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:01.654642 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:01.654645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.655517 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.655518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:01.655614 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:01.655619 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:08:01.655688 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 10:08:01.655739 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":68}
I20260925 10:08:01.655848 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:01.651804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:33421 (local address 127.14.41.67:34819)
0925 10:08:01.652039 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:01.652048 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:01.652104 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:01.652285 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:01.652286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:01.652288 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:01.652347 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:01.652348 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.653260 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.653261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:01.654642 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:01.654645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.655517 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.655518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:01.655614 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:01.655619 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:08:01.655688 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 10:08:01.655739 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":132,"thread_start_us":117,"threads_started":1}
I20260925 10:08:01.655824 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:01.651902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:34819 (local address 127.14.41.65:33421)
0925 10:08:01.651968 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:01.651971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:01.652147 (+ 176us) server_negotiation.cc:407] Connection header received
0925 10:08:01.652180 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:01.652181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:01.652196 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:01.652255 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:01.652425 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.653133 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.654754 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.655403 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.655479 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:01.655488 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:01.655712 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 10:08:01.655750 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:08:01.656263 15216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1b680ffa0a4dac92e144c2a180bc3f" candidate_uuid: "6199c95b40964270b72e7276d5e25a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9f09b7978d648ffa146014037447381" is_pre_election: true
I20260925 10:08:01.656319 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1b680ffa0a4dac92e144c2a180bc3f" candidate_uuid: "6199c95b40964270b72e7276d5e25a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87571b48e90e4003b644007f6e0eb468" is_pre_election: true
I20260925 10:08:01.656458 15216 raft_consensus.cc:2454] T 4b1b680ffa0a4dac92e144c2a180bc3f P f9f09b7978d648ffa146014037447381 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6199c95b40964270b72e7276d5e25a6d in term 0.
I20260925 10:08:01.656575 15351 raft_consensus.cc:2454] T 4b1b680ffa0a4dac92e144c2a180bc3f P 87571b48e90e4003b644007f6e0eb468 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6199c95b40964270b72e7276d5e25a6d in term 0.
I20260925 10:08:01.656745 15422 leader_election.cc:302] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6199c95b40964270b72e7276d5e25a6d, f9f09b7978d648ffa146014037447381; no voters:
I20260925 10:08:01.656890 15558 raft_consensus.cc:2781] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:01.656950 15558 raft_consensus.cc:482] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:01.656975 15558 raft_consensus.cc:3037] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:01.658155 15558 raft_consensus.cc:504] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6199c95b40964270b72e7276d5e25a6d" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 38615 } } peers { permanent_uuid: "87571b48e90e4003b644007f6e0eb468" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 36997 } } peers { permanent_uuid: "f9f09b7978d648ffa146014037447381" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 33421 } }
I20260925 10:08:01.658319 15558 leader_election.cc:288] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d [CANDIDATE]: Term 1 election: Requested vote from peers 87571b48e90e4003b644007f6e0eb468 (127.14.41.66:36997), f9f09b7978d648ffa146014037447381 (127.14.41.65:33421)
I20260925 10:08:01.658577 15216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1b680ffa0a4dac92e144c2a180bc3f" candidate_uuid: "6199c95b40964270b72e7276d5e25a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9f09b7978d648ffa146014037447381"
I20260925 10:08:01.658669 15216 raft_consensus.cc:3037] T 4b1b680ffa0a4dac92e144c2a180bc3f P f9f09b7978d648ffa146014037447381 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:01.658671 15351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1b680ffa0a4dac92e144c2a180bc3f" candidate_uuid: "6199c95b40964270b72e7276d5e25a6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87571b48e90e4003b644007f6e0eb468"
I20260925 10:08:01.658771 15351 raft_consensus.cc:3037] T 4b1b680ffa0a4dac92e144c2a180bc3f P 87571b48e90e4003b644007f6e0eb468 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:01.659508 15216 raft_consensus.cc:2454] T 4b1b680ffa0a4dac92e144c2a180bc3f P f9f09b7978d648ffa146014037447381 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6199c95b40964270b72e7276d5e25a6d in term 1.
I20260925 10:08:01.659601 15351 raft_consensus.cc:2454] T 4b1b680ffa0a4dac92e144c2a180bc3f P 87571b48e90e4003b644007f6e0eb468 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6199c95b40964270b72e7276d5e25a6d in term 1.
I20260925 10:08:01.659696 15422 leader_election.cc:302] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6199c95b40964270b72e7276d5e25a6d, f9f09b7978d648ffa146014037447381; no voters:
I20260925 10:08:01.659837 15558 raft_consensus.cc:2781] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:01.660032 15558 raft_consensus.cc:686] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d [term 1 LEADER]: Becoming Leader. State: Replica: 6199c95b40964270b72e7276d5e25a6d, State: Running, Role: LEADER
I20260925 10:08:01.660147 15558 consensus_queue.cc:237] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6199c95b40964270b72e7276d5e25a6d" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 38615 } } peers { permanent_uuid: "87571b48e90e4003b644007f6e0eb468" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 36997 } } peers { permanent_uuid: "f9f09b7978d648ffa146014037447381" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 33421 } }
I20260925 10:08:01.661015 15071 catalog_manager.cc:5495] T 4b1b680ffa0a4dac92e144c2a180bc3f P 6199c95b40964270b72e7276d5e25a6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6199c95b40964270b72e7276d5e25a6d (127.14.41.67). New cstate: current_term: 1 leader_uuid: "6199c95b40964270b72e7276d5e25a6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6199c95b40964270b72e7276d5e25a6d" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 38615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87571b48e90e4003b644007f6e0eb468" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 36997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9f09b7978d648ffa146014037447381" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 33421 } health_report { overall_health: UNKNOWN } } }
Sep 25 10:08:01 dist-test-slave-s6tm krb5kdc[15026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.65@KRBTEST.COM
I20260925 10:08:01.722944 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:01.712275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53368 (local address 127.14.41.65:33421)
0925 10:08:01.712365 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:01.712367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:01.712455 (+ 88us) server_negotiation.cc:407] Connection header received
0925 10:08:01.712497 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:01.712498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:01.712508 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:01.712572 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:01.713431 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.714421 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.715674 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.716070 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.716146 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:01.719330 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:01.719345 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:01.719346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:01.719355 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:01.721622 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:01.722058 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:01.722062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:01.722063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:01.722137 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:01.722353 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:01.722354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:01.722354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:01.722524 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:01.722703 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:01.722742 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 10:08:01.722830 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 10:08:01 dist-test-slave-s6tm krb5kdc[15026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330881, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:08:01 dist-test-slave-s6tm krb5kdc[15026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330881, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.41.65@KRBTEST.COM
I20260925 10:08:01.747503 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:01.739250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53382 (local address 127.14.41.65:33421)
0925 10:08:01.739384 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:01.739386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:01.739446 (+ 60us) server_negotiation.cc:407] Connection header received
0925 10:08:01.739525 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:01.739526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:01.739544 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:01.739610 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:01.740382 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.740896 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.741822 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.742028 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.742106 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:01.744646 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:01.744648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:01.744648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:01.744656 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:01.746138 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:01.746532 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:01.746534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:01.746534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:01.746597 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:01.746804 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:01.746805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:01.746805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:01.746914 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:01.747088 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:01.747315 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 10:08:01.747404 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
Sep 25 10:08:01 dist-test-slave-s6tm krb5kdc[15026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:08:01 dist-test-slave-s6tm krb5kdc[15026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.65@KRBTEST.COM
I20260925 10:08:01.772149 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:01.763784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.14.41.65:33421)
0925 10:08:01.763861 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:01.763863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:01.764112 (+ 249us) server_negotiation.cc:407] Connection header received
0925 10:08:01.764198 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:01.764199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:01.764210 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:01.764284 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:01.764997 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.765492 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.766245 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:01.766440 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:01.766527 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:01.769048 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:01.769049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:01.769049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:01.769056 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:01.770719 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:01.771177 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:01.771178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:01.771179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:01.771234 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:01.771498 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:01.771499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:01.771499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:01.771603 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:01.771681 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:01.771923 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 10:08:01.772021 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 10:08:01.772676 15176 tablet_service.cc:2908] Rejecting scan request for tablet 4b1b680ffa0a4dac92e144c2a180bc3f: Uninitialized: clean time has not yet been initialized
I20260925 10:08:01.773659 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 15133
I20260925 10:08:01.778980 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 15267
I20260925 10:08:01.784283 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 15403
I20260925 10:08:01.789804 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 15039
2026-09-25T10:08:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2340 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:08:01 dist-test-slave-s6tm krb5kdc[15574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:08:01 dist-test-slave-s6tm krb5kdc[15574](info): set up 2 sockets
Sep 25 10:08:01 dist-test-slave-s6tm krb5kdc[15574](info): commencing operation
krb5kdc: starting...
W20260925 10:08:02.837086 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:08:02 dist-test-slave-s6tm krb5kdc[15574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:08:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:08:02.967505 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:42637
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:42637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:03.054388 15587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:03.054627 15587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:03.054652 15587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:03.055860 15587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:03.055903 15587 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 10:08:03.055917 15587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:03.056010 15587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:03.056036 15587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:03.056049 15587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:08:03.057592 15587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:42637
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:42637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:03.057802 15587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:03.057996 15587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:08:03.063544 15587 server_base.cc:1034] running on GCE node
W20260925 10:08:03.063501 15592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:03.063520 15595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:03.063557 15593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:03.064395 15587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:03.064790 15587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:03.065994 15587 hybrid_clock.cc:648] HybridClock initialized: now 1790330883065964 us; error 51 us; skew 500 ppm
Sep 25 10:08:03 dist-test-slave-s6tm krb5kdc[15574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330883, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:03.069149 15587 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:08:03.069722 15587 webserver.cc:460] Webserver started at http://127.14.41.126:38003/ using document root <none> and password file <none>
I20260925 10:08:03.069998 15587 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:03.070081 15587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:03.070236 15587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:03.071347 15587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "6352a46aef9b41ac978565508dc8cc0e"
format_stamp: "Formatted at 2026-09-25 10:08:03 on dist-test-slave-s6tm"
server_key: "c9606116822037dc0aeff804913f72df"
server_key_iv: "2a92f0e15e8d0d66c438b91d88062177"
server_key_version: "encryptionkey@0"
I20260925 10:08:03.071749 15587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "6352a46aef9b41ac978565508dc8cc0e"
format_stamp: "Formatted at 2026-09-25 10:08:03 on dist-test-slave-s6tm"
server_key: "c9606116822037dc0aeff804913f72df"
server_key_iv: "2a92f0e15e8d0d66c438b91d88062177"
server_key_version: "encryptionkey@0"
I20260925 10:08:03.073505 15587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:03.074451 15602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:03.074765 15587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:03.074913 15587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "6352a46aef9b41ac978565508dc8cc0e"
format_stamp: "Formatted at 2026-09-25 10:08:03 on dist-test-slave-s6tm"
server_key: "c9606116822037dc0aeff804913f72df"
server_key_iv: "2a92f0e15e8d0d66c438b91d88062177"
server_key_version: "encryptionkey@0"
I20260925 10:08:03.075013 15587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:03.089682 15587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:03.092442 15587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:03.092630 15587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:03.097980 15587 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:42637
I20260925 10:08:03.098160 15666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:42637 every 8 connection(s)
I20260925 10:08:03.098462 15587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:03.099197 15667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:03.101725 15667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e: Bootstrap starting.
I20260925 10:08:03.102353 15667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:03.102699 15667 log.cc:824] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:03.103302 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 15587
I20260925 10:08:03.103412 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:08:03.103626 14501 external_mini_cluster.cc:1428] Setting key e34a4b3ca80a1df620c5d22ebb1558f5
I20260925 10:08:03.103652 15667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e: No bootstrap required, opened a new log
I20260925 10:08:03.106573 15667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6352a46aef9b41ac978565508dc8cc0e" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42637 } }
I20260925 10:08:03.106724 15667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:03.106747 15667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6352a46aef9b41ac978565508dc8cc0e, State: Initialized, Role: FOLLOWER
I20260925 10:08:03.106895 15667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6352a46aef9b41ac978565508dc8cc0e" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42637 } }
I20260925 10:08:03.106976 15667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:08:03.106999 15667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:03.107030 15667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:03.107681 15667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6352a46aef9b41ac978565508dc8cc0e" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42637 } }
I20260925 10:08:03.107817 15667 leader_election.cc:302] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6352a46aef9b41ac978565508dc8cc0e; no voters:
I20260925 10:08:03.108048 15667 leader_election.cc:288] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:08:03.108073 15672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:03.108199 15672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e [term 1 LEADER]: Becoming Leader. State: Replica: 6352a46aef9b41ac978565508dc8cc0e, State: Running, Role: LEADER
I20260925 10:08:03.108304 15672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6352a46aef9b41ac978565508dc8cc0e" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42637 } }
I20260925 10:08:03.108348 15667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:03.108680 15673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6352a46aef9b41ac978565508dc8cc0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6352a46aef9b41ac978565508dc8cc0e" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42637 } } }
I20260925 10:08:03.108767 15673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:03.109020 15674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6352a46aef9b41ac978565508dc8cc0e. Latest consensus state: current_term: 1 leader_uuid: "6352a46aef9b41ac978565508dc8cc0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6352a46aef9b41ac978565508dc8cc0e" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42637 } } }
I20260925 10:08:03.109123 15674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:03.109484 15676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:03.110117 15676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:03.112058 15676 catalog_manager.cc:1347] Generated new cluster ID: 68b43a2b77b8449baad1dd45047c2fe5
I20260925 10:08:03.112115 15676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:08:03 dist-test-slave-s6tm krb5kdc[15574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:03.116981 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.104218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54082 (local address 127.14.41.126:42637)
0925 10:08:03.104424 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:03.104436 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:03.104476 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:08:03.106116 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:03.106127 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:03.106203 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:03.106382 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:03.107151 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.108314 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.109422 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.110169 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.110305 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.113337 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:03.113341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:03.113342 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:03.113364 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:03.115141 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:03.115565 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:03.115568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:03.115569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:03.115663 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:03.115886 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:03.115887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:03.115887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:03.116198 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:03.116388 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:03.116472 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 10:08:03.116827 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260925 10:08:03.119185 15676 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:08:03.119872 15676 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:03.131009 15676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e: Generated new TSK 0
I20260925 10:08:03.131251 15676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
I20260925 10:08:03.160024 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:42637
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:03.244827 15693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:03.245085 15693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:03.245127 15693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:03.246320 15693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:03.246395 15693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:03.246428 15693 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:08:03.246469 15693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:08:03.247980 15693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.41.126:42637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:03.248256 15693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:03.248489 15693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:03.253682 15699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:03.253750 15701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:03.253679 15698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:03.254015 15693 server_base.cc:1034] running on GCE node
I20260925 10:08:03.254210 15693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:03.254420 15693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:03.255559 15693 hybrid_clock.cc:648] HybridClock initialized: now 1790330883255543 us; error 25 us; skew 500 ppm
Sep 25 10:08:03 dist-test-slave-s6tm krb5kdc[15574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330883, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:03.258136 15693 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:08:03.258672 15693 webserver.cc:460] Webserver started at http://127.14.41.65:42565/ using document root <none> and password file <none>
I20260925 10:08:03.258920 15693 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:03.258970 15693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:03.259091 15693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:03.260327 15693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "86ce3c8a927340d78c19f0c5e1c386d9"
format_stamp: "Formatted at 2026-09-25 10:08:03 on dist-test-slave-s6tm"
server_key: "aef8f8c5f0e2b86684c1e548693a966f"
server_key_iv: "2bb266714f4e18c60458891ba9a88ce6"
server_key_version: "encryptionkey@0"
I20260925 10:08:03.260641 15693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "86ce3c8a927340d78c19f0c5e1c386d9"
format_stamp: "Formatted at 2026-09-25 10:08:03 on dist-test-slave-s6tm"
server_key: "aef8f8c5f0e2b86684c1e548693a966f"
server_key_iv: "2bb266714f4e18c60458891ba9a88ce6"
server_key_version: "encryptionkey@0"
I20260925 10:08:03.261976 15693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:08:03.262706 15708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:03.262956 15693 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:03.263067 15693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "86ce3c8a927340d78c19f0c5e1c386d9"
format_stamp: "Formatted at 2026-09-25 10:08:03 on dist-test-slave-s6tm"
server_key: "aef8f8c5f0e2b86684c1e548693a966f"
server_key_iv: "2bb266714f4e18c60458891ba9a88ce6"
server_key_version: "encryptionkey@0"
I20260925 10:08:03.263146 15693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:03.279664 15693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:03.281982 15693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:03.282157 15693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:03.282694 15693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:03.282761 15693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:03.282812 15693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:03.282836 15693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:03.290254 15693 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:46701
I20260925 10:08:03.290424 15824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:46701 every 8 connection(s)
Sep 25 10:08:03 dist-test-slave-s6tm krb5kdc[15574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330883, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:03.290788 15693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:08:03.294571 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.283272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:49257 (local address 127.14.41.126:42637)
0925 10:08:03.283319 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:03.283321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:03.283577 (+ 256us) server_negotiation.cc:407] Connection header received
0925 10:08:03.284242 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:03.284244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:03.284260 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:03.284336 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:03.285022 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.286049 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.287743 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.288128 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.288213 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.291374 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:03.291376 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:03.291377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:03.291386 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:03.293144 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:03.293562 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:03.293563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:03.293564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:03.293621 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:03.293843 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:03.293843 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:03.293844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:03.293965 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:03.294021 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:03.294410 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 10:08:03.294503 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:08:03.294623 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.283275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:42637 (local address 127.14.41.65:49257)
0925 10:08:03.283457 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:03.283493 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:03.284101 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:03.284469 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:03.284474 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:03.284718 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:03.284944 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.284949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.286154 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.286155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.287614 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.287617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.288224 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.288225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.288330 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.288954 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:03.288966 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:03.291224 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:03.293232 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:03.293239 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:03.293241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:03.293483 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:03.293684 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:03.293684 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:03.293685 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:03.293784 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:03.294052 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:03.294058 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:03.294207 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 10:08:03.294310 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 10:08:03.294448 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
I20260925 10:08:03.295257 15825 heartbeater.cc:347] Connected to a master server at 127.14.41.126:42637
I20260925 10:08:03.295356 15825 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:03.295591 15825 heartbeater.cc:511] Master 127.14.41.126:42637 requested a full tablet report, sending...
I20260925 10:08:03.295910 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 15693
I20260925 10:08:03.296044 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:08:03.296200 15619 ts_manager.cc:194] Registered new tserver with Master: 86ce3c8a927340d78c19f0c5e1c386d9 (127.14.41.65:46701)
I20260925 10:08:03.296252 14501 external_mini_cluster.cc:1428] Setting key 84d2d2efdac8924caeebcf624310bc45
I20260925 10:08:03.297190 15619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:49257
I20260925 10:08:03.303543 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.296419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:42637 (local address 127.14.41.65:58451)
0925 10:08:03.296729 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:03.296739 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:03.296815 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:03.296973 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:03.296975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:03.297189 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:03.297324 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.297327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.298258 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.298259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.299127 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.299130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.299521 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.299522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.299594 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.300111 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:03.300115 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:03.300430 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:03.302378 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:03.302380 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:03.302380 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:03.302565 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:03.302863 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:03.302864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:03.302864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:03.302935 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:03.303251 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:03.303252 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:03.303297 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 10:08:03.303356 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 10:08:03.303454 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":100,"threads_started":1}
I20260925 10:08:03.303604 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.296634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:58451 (local address 127.14.41.126:42637)
0925 10:08:03.296673 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:03.296675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:03.296797 (+ 122us) server_negotiation.cc:407] Connection header received
0925 10:08:03.296867 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:03.296868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:03.296879 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:03.296926 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:03.297421 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.298148 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.299237 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.299428 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.299490 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.300524 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:03.300526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:03.300527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:03.300535 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:03.302281 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:03.302644 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:03.302645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:03.302645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:03.302712 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:03.303018 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:03.303019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:03.303019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:03.303176 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:03.303258 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:03.303404 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 10:08:03.303496 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.14.41.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
I20260925 10:08:03.324824 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:42637
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:03.418269 15834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:03.418603 15834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:03.418648 15834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:03.420065 15834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:03.420152 15834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:03.420192 15834 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:08:03.420249 15834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:08:03.421974 15834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.41.126:42637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:03.422277 15834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:03.422552 15834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:03.427917 15842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:03.427908 15839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:03.428099 15840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:03.428256 15834 server_base.cc:1034] running on GCE node
I20260925 10:08:03.428509 15834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:03.428774 15834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:03.429983 15834 hybrid_clock.cc:648] HybridClock initialized: now 1790330883429970 us; error 48 us; skew 500 ppm
Sep 25 10:08:03 dist-test-slave-s6tm krb5kdc[15574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330883, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:03.433091 15834 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:08:03.433614 15834 webserver.cc:460] Webserver started at http://127.14.41.66:46483/ using document root <none> and password file <none>
I20260925 10:08:03.433877 15834 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:03.433954 15834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:03.434115 15834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:03.435324 15834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "08d887e583964f3e9724a9cf0d5837e4"
format_stamp: "Formatted at 2026-09-25 10:08:03 on dist-test-slave-s6tm"
server_key: "7c97b2070b11d14abd71805723285a7b"
server_key_iv: "c0611477f690a30a34608dce0d54e5ee"
server_key_version: "encryptionkey@0"
I20260925 10:08:03.435722 15834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "08d887e583964f3e9724a9cf0d5837e4"
format_stamp: "Formatted at 2026-09-25 10:08:03 on dist-test-slave-s6tm"
server_key: "7c97b2070b11d14abd71805723285a7b"
server_key_iv: "c0611477f690a30a34608dce0d54e5ee"
server_key_version: "encryptionkey@0"
I20260925 10:08:03.437263 15834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:03.438180 15849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:03.438354 15834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:03.438488 15834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "08d887e583964f3e9724a9cf0d5837e4"
format_stamp: "Formatted at 2026-09-25 10:08:03 on dist-test-slave-s6tm"
server_key: "7c97b2070b11d14abd71805723285a7b"
server_key_iv: "c0611477f690a30a34608dce0d54e5ee"
server_key_version: "encryptionkey@0"
I20260925 10:08:03.438571 15834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:03.447595 15834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:03.450142 15834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:03.450304 15834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:03.450819 15834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:03.450884 15834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:03.450933 15834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:03.450961 15834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:03.458204 15834 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:45437
I20260925 10:08:03.458356 15965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:45437 every 8 connection(s)
I20260925 10:08:03.458680 15834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:08:03 dist-test-slave-s6tm krb5kdc[15574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330883, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:03.460263 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 15834
I20260925 10:08:03.460361 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:08:03.460558 14501 external_mini_cluster.cc:1428] Setting key 56bd982d213bfb60975baa7d09027051
I20260925 10:08:03.462955 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.451408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:45505 (local address 127.14.41.126:42637)
0925 10:08:03.451459 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:03.451461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:03.451672 (+ 211us) server_negotiation.cc:407] Connection header received
0925 10:08:03.452352 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:03.452353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:03.452367 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:03.452426 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:03.453345 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.454255 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.455999 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.456342 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.456437 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.459398 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:03.459400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:03.459400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:03.459411 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:03.461436 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:03.461892 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:03.461896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:03.461896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:03.461957 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:03.462247 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:03.462248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:03.462248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:03.462380 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:03.462455 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:03.462760 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 10:08:03.462861 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:08:03.463171 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.451362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:42637 (local address 127.14.41.66:45505)
0925 10:08:03.451550 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:03.451586 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:03.452233 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:03.452507 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:03.452511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:03.452918 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:03.453229 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.453236 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.454375 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.454376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.455856 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.455861 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.456449 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.456450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.456594 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.457274 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:03.457285 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:03.459264 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:03.461524 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:03.461533 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:03.461536 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:03.461790 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:03.462032 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:03.462033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:03.462034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:03.462166 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:03.462457 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:03.462465 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:03.462618 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 10:08:03.462728 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 10:08:03.462907 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":93,"threads_started":1}
I20260925 10:08:03.463670 15966 heartbeater.cc:347] Connected to a master server at 127.14.41.126:42637
I20260925 10:08:03.463765 15966 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:03.463996 15966 heartbeater.cc:511] Master 127.14.41.126:42637 requested a full tablet report, sending...
I20260925 10:08:03.464776 15618 ts_manager.cc:194] Registered new tserver with Master: 08d887e583964f3e9724a9cf0d5837e4 (127.14.41.66:45437)
I20260925 10:08:03.465314 15618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:45505
WARNING: no policy specified for kudu/127.14.41.67@KRBTEST.COM; defaulting to no policy
I20260925 10:08:03.472395 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.464812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:42637 (local address 127.14.41.66:49519)
0925 10:08:03.465079 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:03.465097 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:03.465168 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:03.465299 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:03.465300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:03.465541 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:03.465614 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.465617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.466574 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.466575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.467542 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.467545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.468103 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.468105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.468199 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.468738 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:03.468744 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:03.469176 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:03.470969 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:03.470970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:03.470971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:03.471163 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:03.471428 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:03.471429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:03.471430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:03.471499 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:03.472080 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:03.472082 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:03.472128 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 10:08:03.472194 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 10:08:03.472285 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":182,"threads_started":1}
I20260925 10:08:03.472419 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.464890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:49519 (local address 127.14.41.126:42637)
0925 10:08:03.464937 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:03.464939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:03.465174 (+ 235us) server_negotiation.cc:407] Connection header received
0925 10:08:03.465224 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:03.465224 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:03.465234 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:03.465294 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:03.465693 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.466470 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.467645 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.467845 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.468096 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.469303 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:03.469305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:03.469305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:03.469314 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:03.470834 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:03.471251 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:03.471253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:03.471253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:03.471313 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:03.471601 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:03.471602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:03.471603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:03.471770 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:03.471932 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:03.472251 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 10:08:03.472352 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
I20260925 10:08:03.492197 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:42637
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:03.575913 15975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:03.576151 15975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:03.576175 15975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:03.577493 15975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:03.577539 15975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:03.577560 15975 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:08:03.577579 15975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:08:03.579133 15975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.41.126:42637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:03.579336 15975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:03.579521 15975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:03.584864 15980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:03.584872 15981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:03.584889 15983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:03.585341 15975 server_base.cc:1034] running on GCE node
I20260925 10:08:03.585570 15975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:03.585839 15975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:03.587014 15975 hybrid_clock.cc:648] HybridClock initialized: now 1790330883586988 us; error 43 us; skew 500 ppm
Sep 25 10:08:03 dist-test-slave-s6tm krb5kdc[15574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330883, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:03.589952 15975 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:08:03.590418 15975 webserver.cc:460] Webserver started at http://127.14.41.67:38777/ using document root <none> and password file <none>
I20260925 10:08:03.590658 15975 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:03.590727 15975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:03.590881 15975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:03.592118 15975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "42ff1943cf664ea682ad77b1709cda24"
format_stamp: "Formatted at 2026-09-25 10:08:03 on dist-test-slave-s6tm"
server_key: "bea4540ce8bc551c4eed186a771380e8"
server_key_iv: "703789adb76f86ab3819b11822edd32f"
server_key_version: "encryptionkey@0"
I20260925 10:08:03.592519 15975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "42ff1943cf664ea682ad77b1709cda24"
format_stamp: "Formatted at 2026-09-25 10:08:03 on dist-test-slave-s6tm"
server_key: "bea4540ce8bc551c4eed186a771380e8"
server_key_iv: "703789adb76f86ab3819b11822edd32f"
server_key_version: "encryptionkey@0"
I20260925 10:08:03.594022 15975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:08:03.594825 15990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:03.595072 15975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:03.595181 15975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "42ff1943cf664ea682ad77b1709cda24"
format_stamp: "Formatted at 2026-09-25 10:08:03 on dist-test-slave-s6tm"
server_key: "bea4540ce8bc551c4eed186a771380e8"
server_key_iv: "703789adb76f86ab3819b11822edd32f"
server_key_version: "encryptionkey@0"
I20260925 10:08:03.595259 15975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:03.625334 15975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:03.627594 15975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:03.627794 15975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:03.628404 15975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:03.628476 15975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:03.628521 15975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:03.628549 15975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:08:03 dist-test-slave-s6tm krb5kdc[15574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330883, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:03.635962 15975 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:33801
I20260925 10:08:03.636027 16107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:33801 every 8 connection(s)
I20260925 10:08:03.636404 15975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:08:03.637974 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 15975
I20260925 10:08:03.638046 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:08:03.638240 14501 external_mini_cluster.cc:1428] Setting key 948e7e26c2967f3664c732405d39aac2
I20260925 10:08:03.638775 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.628969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:53969 (local address 127.14.41.126:42637)
0925 10:08:03.629025 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:03.629027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:03.629246 (+ 219us) server_negotiation.cc:407] Connection header received
0925 10:08:03.629745 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:03.629747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:03.629762 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:03.629838 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:03.630724 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.631645 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.632740 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.633117 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.633204 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.635481 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:03.635483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:03.635483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:03.635491 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:03.637229 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:03.637684 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:03.637685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:03.637685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:03.637738 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:03.638009 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:03.638010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:03.638011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:03.638177 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:03.638265 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:03.638601 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 10:08:03.638683 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:08:03.638952 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.628984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:42637 (local address 127.14.41.67:53969)
0925 10:08:03.629138 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:03.629175 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:03.629645 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:03.629896 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:03.629899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:03.630264 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:03.630616 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.630625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.631777 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.631778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.632670 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.632672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.633244 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.633246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.633402 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.633980 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:03.633988 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:03.635376 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:03.637302 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:03.637307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:03.637308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:03.637618 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:03.637800 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:03.637800 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:03.637801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:03.637929 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:03.638256 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:03.638263 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:03.638454 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 10:08:03.638576 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 10:08:03.638735 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260925 10:08:03.639413 16108 heartbeater.cc:347] Connected to a master server at 127.14.41.126:42637
I20260925 10:08:03.639493 16108 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:03.639699 16108 heartbeater.cc:511] Master 127.14.41.126:42637 requested a full tablet report, sending...
I20260925 10:08:03.640112 15618 ts_manager.cc:194] Registered new tserver with Master: 42ff1943cf664ea682ad77b1709cda24 (127.14.41.67:33801)
I20260925 10:08:03.640516 15618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:53969
I20260925 10:08:03.646155 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.640789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:42637 (local address 127.14.41.67:57659)
0925 10:08:03.640978 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:03.640988 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:03.641046 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:03.641168 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:03.641169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:03.641329 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:03.641388 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.641390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.642082 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.642083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.642677 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.642680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.643052 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.643053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.643113 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.643505 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:03.643509 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:03.643791 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:03.645256 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:03.645257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:03.645257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:03.645455 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:03.645620 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:03.645621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:03.645621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:03.645672 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:03.645931 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:03.645932 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:03.645966 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 10:08:03.646015 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 10:08:03.646088 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":80,"threads_started":1}
I20260925 10:08:03.646155 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.640874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:57659 (local address 127.14.41.126:42637)
0925 10:08:03.640916 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:03.640918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:03.641058 (+ 140us) server_negotiation.cc:407] Connection header received
0925 10:08:03.641084 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:03.641085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:03.641095 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:03.641166 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:03.641470 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.641989 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.642762 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.642981 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.643038 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.643895 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:03.643896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:03.643896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:03.643903 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:03.645180 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:03.645517 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:03.645518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:03.645518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:03.645564 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:03.645723 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:03.645724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:03.645724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:03.645815 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:03.645929 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:03.646035 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 10:08:03.646092 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:08:03.650377 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:08:03 dist-test-slave-s6tm krb5kdc[15574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330883, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:08:03 dist-test-slave-s6tm krb5kdc[15574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330883, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:03.674727 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.667353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54086 (local address 127.14.41.126:42637)
0925 10:08:03.667408 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:03.667410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:03.667417 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:08:03.667496 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:03.667497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:03.667506 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:03.667560 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:03.668145 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.668678 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.669434 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.669634 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.669700 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.672166 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:03.672168 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:03.672168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:03.672176 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:03.673669 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:03.674025 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:03.674026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:03.674027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:03.674089 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:03.674253 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:03.674253 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:03.674253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:03.674354 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:03.674426 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:03.674530 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 10:08:03.674595 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:08:03.676713 15618 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:08:03.678428 15618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:08:03.685928 16126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.681612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:46701 (local address 127.0.0.1:50734)
0925 10:08:03.681795 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:03.681805 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:03.681865 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:03.682377 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:03.682378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:03.682381 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:03.682429 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.682430 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.683265 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.683266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.684537 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.684539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.685606 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.685607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.685691 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.685709 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 10:08:03.685783 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 10:08:03.685841 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
I20260925 10:08:03.686100 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.681449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.14.41.67:33801)
0925 10:08:03.682005 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:03.682009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:03.682019 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:08:03.682056 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:03.682058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:03.682073 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:03.682303 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:03.682369 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.683508 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.684913 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.685632 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.685756 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.685826 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:03.685969 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 10:08:03.686006 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":108,"threads_started":1}
I20260925 10:08:03.686507 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.681854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50734 (local address 127.14.41.65:46701)
0925 10:08:03.682143 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:03.682148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:03.682157 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:08:03.682198 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:03.682200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:03.682216 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:03.682383 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:03.682488 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.683181 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.684649 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.685512 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.685602 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.685671 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:03.686323 (+ 652us) server_negotiation.cc:299] Negotiation successful
0925 10:08:03.686373 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":122,"threads_started":1}
I20260925 10:08:03.686082 16125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.681245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:33801 (local address 127.0.0.1:45120)
0925 10:08:03.681403 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:03.681453 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:03.681533 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:03.682180 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:03.682180 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:03.682182 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:03.682230 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.682231 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.683601 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.683602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.684691 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.684692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.685771 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.685772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.685851 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.685854 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:03.685950 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 10:08:03.685996 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":43,"threads_started":1}
I20260925 10:08:03.686810 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.681440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:45437 (local address 127.0.0.1:46554)
0925 10:08:03.681745 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:03.681756 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:03.681821 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:03.682102 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:03.682105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:03.682107 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:03.682189 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.682190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.684758 (+ 2568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.684758 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.685418 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.685419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.686474 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.686476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.686567 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.686570 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:03.686685 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 10:08:03.686732 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":86,"threads_started":1}
I20260925 10:08:03.687063 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.681574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46554 (local address 127.14.41.66:45437)
0925 10:08:03.681766 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:03.681772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:03.681809 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:08:03.681902 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:03.681904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:03.681922 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:03.681997 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:03.682277 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.683521 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.685504 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.686288 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.686366 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.686423 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:03.686914 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 10:08:03.686963 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":120,"threads_started":1}
I20260925 10:08:03.687491 16041 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6ea2c78d4d4256ac53dacf8f346557 (DEFAULT_TABLE table=test-table [id=8692376b28fa418b8d1a195d9494135c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:03.687492 15759 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6ea2c78d4d4256ac53dacf8f346557 (DEFAULT_TABLE table=test-table [id=8692376b28fa418b8d1a195d9494135c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:03.687745 15900 tablet_service.cc:1467] Processing CreateTablet for tablet 7c6ea2c78d4d4256ac53dacf8f346557 (DEFAULT_TABLE table=test-table [id=8692376b28fa418b8d1a195d9494135c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:03.687855 15759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6ea2c78d4d4256ac53dacf8f346557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:03.687856 16041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6ea2c78d4d4256ac53dacf8f346557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:03.688120 15900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c6ea2c78d4d4256ac53dacf8f346557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:03.691424 16131 tablet_bootstrap.cc:492] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4: Bootstrap starting.
I20260925 10:08:03.691679 16133 tablet_bootstrap.cc:492] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9: Bootstrap starting.
I20260925 10:08:03.692103 16132 tablet_bootstrap.cc:492] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24: Bootstrap starting.
I20260925 10:08:03.692184 16131 tablet_bootstrap.cc:654] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:03.692291 16133 tablet_bootstrap.cc:654] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:03.692510 16131 log.cc:824] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:03.692691 16133 log.cc:824] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:03.693148 16132 tablet_bootstrap.cc:654] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:03.693393 16131 tablet_bootstrap.cc:492] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4: No bootstrap required, opened a new log
I20260925 10:08:03.693475 16131 ts_tablet_manager.cc:1397] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:08:03.693444 16132 log.cc:824] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:03.694170 16132 tablet_bootstrap.cc:492] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24: No bootstrap required, opened a new log
I20260925 10:08:03.694222 16132 ts_tablet_manager.cc:1397] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:08:03.695245 16131 raft_consensus.cc:348] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } }
I20260925 10:08:03.695416 16131 raft_consensus.cc:374] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:03.695459 16131 raft_consensus.cc:729] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d887e583964f3e9724a9cf0d5837e4, State: Initialized, Role: FOLLOWER
I20260925 10:08:03.695582 16131 consensus_queue.cc:260] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } }
I20260925 10:08:03.695856 16131 ts_tablet_manager.cc:1428] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:08:03.695914 15966 heartbeater.cc:503] Master 127.14.41.126:42637 was elected leader, sending a full tablet report...
I20260925 10:08:03.695917 16132 raft_consensus.cc:348] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } }
I20260925 10:08:03.696105 16132 raft_consensus.cc:374] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:03.696147 16132 raft_consensus.cc:729] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42ff1943cf664ea682ad77b1709cda24, State: Initialized, Role: FOLLOWER
I20260925 10:08:03.696367 16132 consensus_queue.cc:260] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } }
I20260925 10:08:03.696534 16133 tablet_bootstrap.cc:492] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9: No bootstrap required, opened a new log
I20260925 10:08:03.696609 16133 ts_tablet_manager.cc:1397] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 10:08:03.696635 16132 ts_tablet_manager.cc:1428] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 10:08:03.696770 16108 heartbeater.cc:503] Master 127.14.41.126:42637 was elected leader, sending a full tablet report...
I20260925 10:08:03.698442 16133 raft_consensus.cc:348] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } }
I20260925 10:08:03.698616 16133 raft_consensus.cc:374] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:03.698665 16133 raft_consensus.cc:729] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ce3c8a927340d78c19f0c5e1c386d9, State: Initialized, Role: FOLLOWER
I20260925 10:08:03.698784 16133 consensus_queue.cc:260] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } }
I20260925 10:08:03.699074 16133 ts_tablet_manager.cc:1428] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 10:08:03.699115 15825 heartbeater.cc:503] Master 127.14.41.126:42637 was elected leader, sending a full tablet report...
W20260925 10:08:03.709506 15967 tablet.cc:2369] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:08:03.739171 16139 raft_consensus.cc:482] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:03.739351 16139 raft_consensus.cc:504] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } }
I20260925 10:08:03.739787 16139 leader_election.cc:288] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42ff1943cf664ea682ad77b1709cda24 (127.14.41.67:33801), 08d887e583964f3e9724a9cf0d5837e4 (127.14.41.66:45437)
I20260925 10:08:03.743674 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.740407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:44419 (local address 127.14.41.67:33801)
0925 10:08:03.740493 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:03.740495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:03.740517 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:08:03.740547 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:03.740549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:03.740559 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:03.740610 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:03.740809 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.741587 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.742882 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.743294 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.743352 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.743387 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:03.743551 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 10:08:03.743600 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:08:03.743680 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.740032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:33801 (local address 127.14.41.65:44419)
0925 10:08:03.740080 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:03.740092 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:03.740146 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:03.740648 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:03.740649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:03.740652 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:03.740729 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.740731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.741705 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.741706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.742785 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.742788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.743375 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.743376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.743474 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.743477 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:03.743528 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 10:08:03.743562 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 10:08:03.743737 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.740561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:54955 (local address 127.14.41.66:45437)
0925 10:08:03.740611 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:03.740614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:03.740699 (+ 85us) server_negotiation.cc:407] Connection header received
0925 10:08:03.740786 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:03.740801 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:03.740813 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:03.740892 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:03.741044 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.741704 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.742943 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.743301 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.743394 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.743403 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:03.743618 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 10:08:03.743661 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 10:08:03.743765 16140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.740123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:45437 (local address 127.14.41.65:54955)
0925 10:08:03.740304 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:03.740654 (+ 350us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:03.740722 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:03.740905 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:03.740906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:03.740908 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:03.740966 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.740967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.741809 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.741810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.742858 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.742861 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.743408 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.743409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.743536 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.743541 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:08:03.743598 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 10:08:03.743634 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":107,"threads_started":1}
I20260925 10:08:03.744047 16061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6ea2c78d4d4256ac53dacf8f346557" candidate_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42ff1943cf664ea682ad77b1709cda24" is_pre_election: true
I20260925 10:08:03.744204 16061 raft_consensus.cc:2454] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86ce3c8a927340d78c19f0c5e1c386d9 in term 0.
I20260925 10:08:03.744207 15920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6ea2c78d4d4256ac53dacf8f346557" candidate_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d887e583964f3e9724a9cf0d5837e4" is_pre_election: true
I20260925 10:08:03.744381 15920 raft_consensus.cc:2454] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86ce3c8a927340d78c19f0c5e1c386d9 in term 0.
I20260925 10:08:03.744432 15710 leader_election.cc:302] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42ff1943cf664ea682ad77b1709cda24, 86ce3c8a927340d78c19f0c5e1c386d9; no voters:
I20260925 10:08:03.744766 16139 raft_consensus.cc:2781] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:03.744848 16139 raft_consensus.cc:482] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:03.744877 16139 raft_consensus.cc:3037] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:03.745638 16139 raft_consensus.cc:504] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } }
I20260925 10:08:03.745788 16139 leader_election.cc:288] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [CANDIDATE]: Term 1 election: Requested vote from peers 42ff1943cf664ea682ad77b1709cda24 (127.14.41.67:33801), 08d887e583964f3e9724a9cf0d5837e4 (127.14.41.66:45437)
I20260925 10:08:03.746019 15920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6ea2c78d4d4256ac53dacf8f346557" candidate_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d887e583964f3e9724a9cf0d5837e4"
I20260925 10:08:03.746021 16061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c6ea2c78d4d4256ac53dacf8f346557" candidate_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42ff1943cf664ea682ad77b1709cda24"
I20260925 10:08:03.746114 15920 raft_consensus.cc:3037] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:03.746114 16061 raft_consensus.cc:3037] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:03.746905 16061 raft_consensus.cc:2454] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86ce3c8a927340d78c19f0c5e1c386d9 in term 1.
I20260925 10:08:03.746975 15920 raft_consensus.cc:2454] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86ce3c8a927340d78c19f0c5e1c386d9 in term 1.
I20260925 10:08:03.747072 15710 leader_election.cc:302] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42ff1943cf664ea682ad77b1709cda24, 86ce3c8a927340d78c19f0c5e1c386d9; no voters:
I20260925 10:08:03.747174 16139 raft_consensus.cc:2781] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:03.747330 16139 raft_consensus.cc:686] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 1 LEADER]: Becoming Leader. State: Replica: 86ce3c8a927340d78c19f0c5e1c386d9, State: Running, Role: LEADER
I20260925 10:08:03.747448 16139 consensus_queue.cc:237] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } }
I20260925 10:08:03.748435 15618 catalog_manager.cc:5495] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86ce3c8a927340d78c19f0c5e1c386d9 (127.14.41.65). New cstate: current_term: 1 leader_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } health_report { overall_health: HEALTHY } } }
I20260925 10:08:03.763738 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.760941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50742 (local address 127.14.41.65:46701)
0925 10:08:03.761015 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:03.761017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:03.761026 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:08:03.761116 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:03.761116 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:03.761134 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:03.761193 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:03.761399 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.761869 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.762761 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.762957 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.763057 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.763293 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:03.763401 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:03.763491 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:03.763594 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 10:08:03.763615 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:08:03.766559 16061 raft_consensus.cc:1261] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24 [term 1 FOLLOWER]: Refusing update from remote peer 86ce3c8a927340d78c19f0c5e1c386d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:03.766532 15920 raft_consensus.cc:1261] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4 [term 1 FOLLOWER]: Refusing update from remote peer 86ce3c8a927340d78c19f0c5e1c386d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:03.767109 16141 consensus_queue.cc:1035] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:03.767266 16139 consensus_queue.cc:1035] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:03.770210 16144 mvcc.cc:205] Tried to move back new op lower bound from 7333195299903873024 to 7333195299830259712. Current Snapshot: MvccSnapshot[applied={T|T < 7333195299903873024}]
I20260925 10:08:03.770529 16145 mvcc.cc:205] Tried to move back new op lower bound from 7333195299903873024 to 7333195299830259712. Current Snapshot: MvccSnapshot[applied={T|T < 7333195299903873024}]
I20260925 10:08:03.770869 16146 mvcc.cc:205] Tried to move back new op lower bound from 7333195299903873024 to 7333195299830259712. Current Snapshot: MvccSnapshot[applied={T|T < 7333195299903873024}]
I20260925 10:08:03.774610 15618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:03.774703 15618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:03.775489 15618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e: Sending DeleteTablet for 3 replicas of tablet 7c6ea2c78d4d4256ac53dacf8f346557
I20260925 10:08:03.775806 16041 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c6ea2c78d4d4256ac53dacf8f346557 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:03 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:45120
I20260925 10:08:03.775856 15900 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c6ea2c78d4d4256ac53dacf8f346557 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:03 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:46554
I20260925 10:08:03.776067 15759 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c6ea2c78d4d4256ac53dacf8f346557 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:03 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:50734
I20260925 10:08:03.776471 16154 tablet_replica.cc:330] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24: stopping tablet replica
I20260925 10:08:03.776589 16154 raft_consensus.cc:2229] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:03.776568 15618 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 10:08:03.776700 16154 raft_consensus.cc:2258] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 10:08:03.776721 15618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:08:03.777205 16154 ts_tablet_manager.cc:1905] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:03.778203 16041 tablet_service.cc:1467] Processing CreateTablet for tablet 2c80ed3bf71d40b18095759cf8b86659 (DEFAULT_TABLE table=test-table [id=e86682cbc11447df86ef180a9cf8f677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:03.778203 15759 tablet_service.cc:1467] Processing CreateTablet for tablet 2c80ed3bf71d40b18095759cf8b86659 (DEFAULT_TABLE table=test-table [id=e86682cbc11447df86ef180a9cf8f677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:03.778258 15900 tablet_service.cc:1467] Processing CreateTablet for tablet 2c80ed3bf71d40b18095759cf8b86659 (DEFAULT_TABLE table=test-table [id=e86682cbc11447df86ef180a9cf8f677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:03.778316 15759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c80ed3bf71d40b18095759cf8b86659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:03.778316 16041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c80ed3bf71d40b18095759cf8b86659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:03.778386 15900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c80ed3bf71d40b18095759cf8b86659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:03.777189 16155 tablet_replica.cc:330] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4: stopping tablet replica
I20260925 10:08:03.779663 16132 tablet_bootstrap.cc:492] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24: Bootstrap starting.
I20260925 10:08:03.779753 16155 raft_consensus.cc:2229] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:03.779812 16131 tablet_bootstrap.cc:492] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4: Bootstrap starting.
I20260925 10:08:03.779886 16155 raft_consensus.cc:2258] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:03.780192 16132 tablet_bootstrap.cc:654] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:03.780260 16131 tablet_bootstrap.cc:654] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:03.780426 16155 ts_tablet_manager.cc:1905] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:03.780503 16133 tablet_bootstrap.cc:492] T 2c80ed3bf71d40b18095759cf8b86659 P 86ce3c8a927340d78c19f0c5e1c386d9: Bootstrap starting.
I20260925 10:08:03.780936 16133 tablet_bootstrap.cc:654] T 2c80ed3bf71d40b18095759cf8b86659 P 86ce3c8a927340d78c19f0c5e1c386d9: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:03.781633 16133 tablet_bootstrap.cc:492] T 2c80ed3bf71d40b18095759cf8b86659 P 86ce3c8a927340d78c19f0c5e1c386d9: No bootstrap required, opened a new log
I20260925 10:08:03.781679 16133 ts_tablet_manager.cc:1397] T 2c80ed3bf71d40b18095759cf8b86659 P 86ce3c8a927340d78c19f0c5e1c386d9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:03.781826 16133 raft_consensus.cc:348] T 2c80ed3bf71d40b18095759cf8b86659 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } }
I20260925 10:08:03.781924 16133 raft_consensus.cc:374] T 2c80ed3bf71d40b18095759cf8b86659 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:03.781947 16133 raft_consensus.cc:729] T 2c80ed3bf71d40b18095759cf8b86659 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ce3c8a927340d78c19f0c5e1c386d9, State: Initialized, Role: FOLLOWER
I20260925 10:08:03.781993 16133 consensus_queue.cc:260] T 2c80ed3bf71d40b18095759cf8b86659 P 86ce3c8a927340d78c19f0c5e1c386d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } }
I20260925 10:08:03.781996 16156 tablet_replica.cc:330] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9: stopping tablet replica
I20260925 10:08:03.782091 16133 ts_tablet_manager.cc:1428] T 2c80ed3bf71d40b18095759cf8b86659 P 86ce3c8a927340d78c19f0c5e1c386d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:03.782130 16156 raft_consensus.cc:2229] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:08:03.782270 16156 raft_consensus.cc:2258] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:03.782718 16156 ts_tablet_manager.cc:1905] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:03.782718 16155 ts_tablet_manager.cc:1918] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:08:03.782787 16155 log.cc:1196] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/7c6ea2c78d4d4256ac53dacf8f346557
I20260925 10:08:03.783067 16155 ts_tablet_manager.cc:1939] T 7c6ea2c78d4d4256ac53dacf8f346557 P 08d887e583964f3e9724a9cf0d5837e4: Deleting consensus metadata
I20260925 10:08:03.783461 15604 catalog_manager.cc:4841] TS 08d887e583964f3e9724a9cf0d5837e4 (127.14.41.66:45437): tablet 7c6ea2c78d4d4256ac53dacf8f346557 (table test-table [id=8692376b28fa418b8d1a195d9494135c]) successfully deleted
I20260925 10:08:03.783689 16154 ts_tablet_manager.cc:1918] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:08:03.783769 16154 log.cc:1196] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/7c6ea2c78d4d4256ac53dacf8f346557
I20260925 10:08:03.784075 16154 ts_tablet_manager.cc:1939] T 7c6ea2c78d4d4256ac53dacf8f346557 P 42ff1943cf664ea682ad77b1709cda24: Deleting consensus metadata
I20260925 10:08:03.784379 16131 tablet_bootstrap.cc:492] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4: No bootstrap required, opened a new log
I20260925 10:08:03.784433 16131 ts_tablet_manager.cc:1397] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 10:08:03.784584 16131 raft_consensus.cc:348] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } }
I20260925 10:08:03.784689 16131 raft_consensus.cc:374] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:03.784720 16131 raft_consensus.cc:729] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d887e583964f3e9724a9cf0d5837e4, State: Initialized, Role: FOLLOWER
I20260925 10:08:03.784763 16131 consensus_queue.cc:260] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } }
I20260925 10:08:03.784857 16131 ts_tablet_manager.cc:1428] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:03.784938 16156 ts_tablet_manager.cc:1918] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:08:03.785056 16156 log.cc:1196] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-0/wal/wals/7c6ea2c78d4d4256ac53dacf8f346557
I20260925 10:08:03.785285 16132 tablet_bootstrap.cc:492] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24: No bootstrap required, opened a new log
I20260925 10:08:03.785342 16132 ts_tablet_manager.cc:1397] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260925 10:08:03.785542 16156 ts_tablet_manager.cc:1939] T 7c6ea2c78d4d4256ac53dacf8f346557 P 86ce3c8a927340d78c19f0c5e1c386d9: Deleting consensus metadata
I20260925 10:08:03.785508 16132 raft_consensus.cc:348] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } }
I20260925 10:08:03.785585 16132 raft_consensus.cc:374] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:03.785611 16132 raft_consensus.cc:729] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42ff1943cf664ea682ad77b1709cda24, State: Initialized, Role: FOLLOWER
I20260925 10:08:03.785666 16132 consensus_queue.cc:260] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } }
I20260925 10:08:03.785784 16132 ts_tablet_manager.cc:1428] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:03.786265 15604 catalog_manager.cc:4841] TS 86ce3c8a927340d78c19f0c5e1c386d9 (127.14.41.65:46701): tablet 7c6ea2c78d4d4256ac53dacf8f346557 (table test-table [id=8692376b28fa418b8d1a195d9494135c]) successfully deleted
I20260925 10:08:03.786808 15604 catalog_manager.cc:4841] TS 42ff1943cf664ea682ad77b1709cda24 (127.14.41.67:33801): tablet 7c6ea2c78d4d4256ac53dacf8f346557 (table test-table [id=8692376b28fa418b8d1a195d9494135c]) successfully deleted
W20260925 10:08:03.887151 16109 tablet.cc:2369] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:08:03.953051 16137 raft_consensus.cc:482] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:03.953184 16137 raft_consensus.cc:504] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } }
I20260925 10:08:03.953622 16137 leader_election.cc:288] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86ce3c8a927340d78c19f0c5e1c386d9 (127.14.41.65:46701), 42ff1943cf664ea682ad77b1709cda24 (127.14.41.67:33801)
I20260925 10:08:03.957758 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.954172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:44247 (local address 127.14.41.67:33801)
0925 10:08:03.954215 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:03.954217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:03.954226 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:08:03.954325 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:03.954326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:03.954338 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:03.954370 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:03.954631 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.955348 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.956901 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.957372 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.957473 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.957482 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:03.957649 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 10:08:03.957688 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:08:03.957773 16160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.953997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:33801 (local address 127.14.41.66:44247)
0925 10:08:03.954134 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:03.954147 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:03.954223 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:03.954443 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:03.954444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:03.954446 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:03.954532 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.954534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.955411 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.955412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.956809 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.956812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.957474 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.957475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.957549 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.957553 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:08:03.957631 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 10:08:03.957676 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260925 10:08:03.958079 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.953878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:46701 (local address 127.14.41.66:35871)
0925 10:08:03.953998 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:03.954014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:03.954091 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:03.954363 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:03.954364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:03.954366 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:03.954449 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.954451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.955386 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.955387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.956866 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:03.956869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.957772 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.957774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:03.957867 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.957870 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:03.957933 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 10:08:03.957975 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":25}
I20260925 10:08:03.958309 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:03.953984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:35871 (local address 127.14.41.65:46701)
0925 10:08:03.954054 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:03.954057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:03.954070 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:08:03.954201 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:03.954204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:03.954218 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:03.954288 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:03.954535 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.955278 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.956994 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:03.957656 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:03.957730 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:03.957738 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:03.958162 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 10:08:03.958216 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:08:03.958534 16061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c80ed3bf71d40b18095759cf8b86659" candidate_uuid: "08d887e583964f3e9724a9cf0d5837e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42ff1943cf664ea682ad77b1709cda24" is_pre_election: true
I20260925 10:08:03.958642 16061 raft_consensus.cc:2454] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d887e583964f3e9724a9cf0d5837e4 in term 0.
I20260925 10:08:03.958597 15779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c80ed3bf71d40b18095759cf8b86659" candidate_uuid: "08d887e583964f3e9724a9cf0d5837e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" is_pre_election: true
I20260925 10:08:03.958774 15779 raft_consensus.cc:2454] T 2c80ed3bf71d40b18095759cf8b86659 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d887e583964f3e9724a9cf0d5837e4 in term 0.
I20260925 10:08:03.958838 15852 leader_election.cc:302] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d887e583964f3e9724a9cf0d5837e4, 42ff1943cf664ea682ad77b1709cda24; no voters:
I20260925 10:08:03.958993 16137 raft_consensus.cc:2781] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:03.959034 16137 raft_consensus.cc:482] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:03.959065 16137 raft_consensus.cc:3037] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:03.960124 16137 raft_consensus.cc:504] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } }
I20260925 10:08:03.960279 16137 leader_election.cc:288] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [CANDIDATE]: Term 1 election: Requested vote from peers 86ce3c8a927340d78c19f0c5e1c386d9 (127.14.41.65:46701), 42ff1943cf664ea682ad77b1709cda24 (127.14.41.67:33801)
I20260925 10:08:03.960510 15779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c80ed3bf71d40b18095759cf8b86659" candidate_uuid: "08d887e583964f3e9724a9cf0d5837e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ce3c8a927340d78c19f0c5e1c386d9"
I20260925 10:08:03.960515 16061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c80ed3bf71d40b18095759cf8b86659" candidate_uuid: "08d887e583964f3e9724a9cf0d5837e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42ff1943cf664ea682ad77b1709cda24"
I20260925 10:08:03.960590 16061 raft_consensus.cc:3037] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:03.960615 15779 raft_consensus.cc:3037] T 2c80ed3bf71d40b18095759cf8b86659 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:03.961295 15779 raft_consensus.cc:2454] T 2c80ed3bf71d40b18095759cf8b86659 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d887e583964f3e9724a9cf0d5837e4 in term 1.
I20260925 10:08:03.961418 16061 raft_consensus.cc:2454] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d887e583964f3e9724a9cf0d5837e4 in term 1.
I20260925 10:08:03.961514 15852 leader_election.cc:302] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d887e583964f3e9724a9cf0d5837e4, 86ce3c8a927340d78c19f0c5e1c386d9; no voters:
I20260925 10:08:03.961642 16137 raft_consensus.cc:2781] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:03.961776 16137 raft_consensus.cc:686] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [term 1 LEADER]: Becoming Leader. State: Replica: 08d887e583964f3e9724a9cf0d5837e4, State: Running, Role: LEADER
I20260925 10:08:03.961854 16137 consensus_queue.cc:237] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } }
I20260925 10:08:03.962713 15619 catalog_manager.cc:5495] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d887e583964f3e9724a9cf0d5837e4 (127.14.41.66). New cstate: current_term: 1 leader_uuid: "08d887e583964f3e9724a9cf0d5837e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } health_report { overall_health: UNKNOWN } } }
Sep 25 10:08:04 dist-test-slave-s6tm krb5kdc[15574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330883, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:04.009505 16168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:04.000768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:42637 (local address 127.0.0.1:54096)
0925 10:08:04.001001 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:04.001012 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:04.001099 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:04.001269 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:04.001270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:04.001627 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:04.001680 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:04.001682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.002730 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.002731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:04.003338 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:04.003341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.003845 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.003846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:04.003897 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:04.004587 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:04.004592 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:04.006105 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:04.008036 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:04.008044 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:04.008046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:04.008260 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:04.008541 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:04.008543 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:04.008543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:04.008633 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:04.009003 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:04.009011 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:04.009134 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 10:08:04.009221 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 10:08:04.009345 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":153,"threads_started":1}
I20260925 10:08:04.009585 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:04.000852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54096 (local address 127.14.41.126:42637)
0925 10:08:04.000923 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:04.000925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:04.001078 (+ 153us) server_negotiation.cc:407] Connection header received
0925 10:08:04.001164 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:04.001165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:04.001176 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:04.001230 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:04.001777 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.002636 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.003429 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.003735 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.003806 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:04.006197 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:04.006199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:04.006199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:04.006208 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:04.007920 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:04.008364 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:04.008366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:04.008366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:04.008433 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:04.008746 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:04.008747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:04.008748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:04.008901 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:04.008982 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:04.009254 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 10:08:04.009415 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":39}
I20260925 10:08:04.012228 15619 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:54096:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 10:08:04.012494 15619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:08:04.013983 15759 tablet_service.cc:1467] Processing CreateTablet for tablet 086ae995f1d54d17abc00acf61630b7c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ca2812c40cc490f9bfc21b0caeb847b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:08:04.014173 15759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086ae995f1d54d17abc00acf61630b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:04.014122 15900 tablet_service.cc:1467] Processing CreateTablet for tablet 086ae995f1d54d17abc00acf61630b7c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ca2812c40cc490f9bfc21b0caeb847b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:08:04.014279 15900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086ae995f1d54d17abc00acf61630b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:04.014102 16041 tablet_service.cc:1467] Processing CreateTablet for tablet 086ae995f1d54d17abc00acf61630b7c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ca2812c40cc490f9bfc21b0caeb847b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:08:04.015151 16041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086ae995f1d54d17abc00acf61630b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:04.015506 16131 tablet_bootstrap.cc:492] T 086ae995f1d54d17abc00acf61630b7c P 08d887e583964f3e9724a9cf0d5837e4: Bootstrap starting.
I20260925 10:08:04.015890 16133 tablet_bootstrap.cc:492] T 086ae995f1d54d17abc00acf61630b7c P 86ce3c8a927340d78c19f0c5e1c386d9: Bootstrap starting.
I20260925 10:08:04.015947 16131 tablet_bootstrap.cc:654] T 086ae995f1d54d17abc00acf61630b7c P 08d887e583964f3e9724a9cf0d5837e4: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:04.016247 16133 tablet_bootstrap.cc:654] T 086ae995f1d54d17abc00acf61630b7c P 86ce3c8a927340d78c19f0c5e1c386d9: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:04.016636 16131 tablet_bootstrap.cc:492] T 086ae995f1d54d17abc00acf61630b7c P 08d887e583964f3e9724a9cf0d5837e4: No bootstrap required, opened a new log
I20260925 10:08:04.016705 16131 ts_tablet_manager.cc:1397] T 086ae995f1d54d17abc00acf61630b7c P 08d887e583964f3e9724a9cf0d5837e4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:04.016840 16132 tablet_bootstrap.cc:492] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24: Bootstrap starting.
I20260925 10:08:04.016860 16131 raft_consensus.cc:348] T 086ae995f1d54d17abc00acf61630b7c P 08d887e583964f3e9724a9cf0d5837e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } }
I20260925 10:08:04.016929 16131 raft_consensus.cc:374] T 086ae995f1d54d17abc00acf61630b7c P 08d887e583964f3e9724a9cf0d5837e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:04.016948 16131 raft_consensus.cc:729] T 086ae995f1d54d17abc00acf61630b7c P 08d887e583964f3e9724a9cf0d5837e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d887e583964f3e9724a9cf0d5837e4, State: Initialized, Role: FOLLOWER
I20260925 10:08:04.016999 16131 consensus_queue.cc:260] T 086ae995f1d54d17abc00acf61630b7c P 08d887e583964f3e9724a9cf0d5837e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } }
I20260925 10:08:04.017102 16131 ts_tablet_manager.cc:1428] T 086ae995f1d54d17abc00acf61630b7c P 08d887e583964f3e9724a9cf0d5837e4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:04.017197 16137 tablet_replica.cc:439] T 086ae995f1d54d17abc00acf61630b7c P 08d887e583964f3e9724a9cf0d5837e4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } } }
I20260925 10:08:04.017297 16137 tablet_replica.cc:442] T 086ae995f1d54d17abc00acf61630b7c P 08d887e583964f3e9724a9cf0d5837e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:08:04.017315 16132 tablet_bootstrap.cc:654] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:04.017867 16133 tablet_bootstrap.cc:492] T 086ae995f1d54d17abc00acf61630b7c P 86ce3c8a927340d78c19f0c5e1c386d9: No bootstrap required, opened a new log
I20260925 10:08:04.017941 16133 ts_tablet_manager.cc:1397] T 086ae995f1d54d17abc00acf61630b7c P 86ce3c8a927340d78c19f0c5e1c386d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:08:04.018074 16132 tablet_bootstrap.cc:492] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24: No bootstrap required, opened a new log
I20260925 10:08:04.018113 16132 ts_tablet_manager.cc:1397] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 10:08:04.018078 16133 raft_consensus.cc:348] T 086ae995f1d54d17abc00acf61630b7c P 86ce3c8a927340d78c19f0c5e1c386d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } }
I20260925 10:08:04.018149 16133 raft_consensus.cc:374] T 086ae995f1d54d17abc00acf61630b7c P 86ce3c8a927340d78c19f0c5e1c386d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:04.018175 16133 raft_consensus.cc:729] T 086ae995f1d54d17abc00acf61630b7c P 86ce3c8a927340d78c19f0c5e1c386d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ce3c8a927340d78c19f0c5e1c386d9, State: Initialized, Role: FOLLOWER
I20260925 10:08:04.018230 16133 consensus_queue.cc:260] T 086ae995f1d54d17abc00acf61630b7c P 86ce3c8a927340d78c19f0c5e1c386d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } }
I20260925 10:08:04.018261 16132 raft_consensus.cc:348] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } }
I20260925 10:08:04.018321 16132 raft_consensus.cc:374] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:04.018319 16133 ts_tablet_manager.cc:1428] T 086ae995f1d54d17abc00acf61630b7c P 86ce3c8a927340d78c19f0c5e1c386d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:04.018342 16132 raft_consensus.cc:729] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42ff1943cf664ea682ad77b1709cda24, State: Initialized, Role: FOLLOWER
I20260925 10:08:04.018397 16132 consensus_queue.cc:260] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } }
I20260925 10:08:04.018355 16149 tablet_replica.cc:439] T 086ae995f1d54d17abc00acf61630b7c P 86ce3c8a927340d78c19f0c5e1c386d9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } } }
I20260925 10:08:04.018517 16149 tablet_replica.cc:442] T 086ae995f1d54d17abc00acf61630b7c P 86ce3c8a927340d78c19f0c5e1c386d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:08:04.018533 16132 ts_tablet_manager.cc:1428] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:04.018608 16138 tablet_replica.cc:439] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } } }
I20260925 10:08:04.018678 16138 tablet_replica.cc:442] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 10:08:04.041965 15826 tablet.cc:2369] T 086ae995f1d54d17abc00acf61630b7c P 86ce3c8a927340d78c19f0c5e1c386d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:08:04.087815 16138 raft_consensus.cc:482] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:04.087922 16138 raft_consensus.cc:504] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } }
I20260925 10:08:04.088336 16138 leader_election.cc:288] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86ce3c8a927340d78c19f0c5e1c386d9 (127.14.41.65:46701), 08d887e583964f3e9724a9cf0d5837e4 (127.14.41.66:45437)
I20260925 10:08:04.091632 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:04.088541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:46701 (local address 127.14.41.67:60333)
0925 10:08:04.088626 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:04.088641 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:04.088712 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:04.088887 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:04.088888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:04.088890 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:04.088992 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:04.088995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.089803 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.089804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:04.090616 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:04.090619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.091341 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.091342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:04.091436 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:04.091439 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:04.091507 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 10:08:04.091551 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 10:08:04.091632 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:04.088633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:60333 (local address 127.14.41.65:46701)
0925 10:08:04.088684 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:04.088686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:04.088697 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:08:04.088794 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:04.088795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:04.088806 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:04.088866 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:04.089116 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.089711 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.090716 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.091261 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.091345 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:04.091355 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:04.091516 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 10:08:04.091564 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 10:08:04.092011 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:04.088745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:47799 (local address 127.14.41.66:45437)
0925 10:08:04.088793 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:04.088796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:04.088971 (+ 175us) server_negotiation.cc:407] Connection header received
0925 10:08:04.089002 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:04.089004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:04.089014 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:04.089077 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:04.089290 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.089783 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.091274 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.091636 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.091723 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:04.091732 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:04.091870 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 10:08:04.091915 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:08:04.092052 16172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:04.088675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:45437 (local address 127.14.41.67:47799)
0925 10:08:04.088865 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:04.088874 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:04.088926 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:04.089117 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:04.089118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:04.089121 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:04.089200 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:04.089201 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.089884 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.089885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:04.091193 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:04.091196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.091723 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.091724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:04.091799 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:04.091803 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:08:04.091863 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 10:08:04.091915 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":1,"thread_start_us":121,"threads_started":1}
I20260925 10:08:04.092109 15779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086ae995f1d54d17abc00acf61630b7c" candidate_uuid: "42ff1943cf664ea682ad77b1709cda24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" is_pre_election: true
I20260925 10:08:04.092226 15779 raft_consensus.cc:2454] T 086ae995f1d54d17abc00acf61630b7c P 86ce3c8a927340d78c19f0c5e1c386d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42ff1943cf664ea682ad77b1709cda24 in term 0.
I20260925 10:08:04.092301 15920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086ae995f1d54d17abc00acf61630b7c" candidate_uuid: "42ff1943cf664ea682ad77b1709cda24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d887e583964f3e9724a9cf0d5837e4" is_pre_election: true
I20260925 10:08:04.092386 15920 raft_consensus.cc:2454] T 086ae995f1d54d17abc00acf61630b7c P 08d887e583964f3e9724a9cf0d5837e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42ff1943cf664ea682ad77b1709cda24 in term 0.
I20260925 10:08:04.092429 15992 leader_election.cc:302] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42ff1943cf664ea682ad77b1709cda24, 86ce3c8a927340d78c19f0c5e1c386d9; no voters:
I20260925 10:08:04.092579 16138 raft_consensus.cc:2781] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:04.092621 16138 raft_consensus.cc:482] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:04.092644 16138 raft_consensus.cc:3037] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:04.093336 16138 raft_consensus.cc:504] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } }
I20260925 10:08:04.093489 16138 leader_election.cc:288] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24 [CANDIDATE]: Term 1 election: Requested vote from peers 86ce3c8a927340d78c19f0c5e1c386d9 (127.14.41.65:46701), 08d887e583964f3e9724a9cf0d5837e4 (127.14.41.66:45437)
I20260925 10:08:04.093679 15779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086ae995f1d54d17abc00acf61630b7c" candidate_uuid: "42ff1943cf664ea682ad77b1709cda24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ce3c8a927340d78c19f0c5e1c386d9"
I20260925 10:08:04.093679 15920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086ae995f1d54d17abc00acf61630b7c" candidate_uuid: "42ff1943cf664ea682ad77b1709cda24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d887e583964f3e9724a9cf0d5837e4"
I20260925 10:08:04.093772 15779 raft_consensus.cc:3037] T 086ae995f1d54d17abc00acf61630b7c P 86ce3c8a927340d78c19f0c5e1c386d9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:04.093772 15920 raft_consensus.cc:3037] T 086ae995f1d54d17abc00acf61630b7c P 08d887e583964f3e9724a9cf0d5837e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:04.094650 15779 raft_consensus.cc:2454] T 086ae995f1d54d17abc00acf61630b7c P 86ce3c8a927340d78c19f0c5e1c386d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42ff1943cf664ea682ad77b1709cda24 in term 1.
I20260925 10:08:04.094650 15920 raft_consensus.cc:2454] T 086ae995f1d54d17abc00acf61630b7c P 08d887e583964f3e9724a9cf0d5837e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42ff1943cf664ea682ad77b1709cda24 in term 1.
I20260925 10:08:04.094951 15992 leader_election.cc:302] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d887e583964f3e9724a9cf0d5837e4, 42ff1943cf664ea682ad77b1709cda24; no voters:
I20260925 10:08:04.095054 16138 raft_consensus.cc:2781] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:04.095239 16138 raft_consensus.cc:686] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24 [term 1 LEADER]: Becoming Leader. State: Replica: 42ff1943cf664ea682ad77b1709cda24, State: Running, Role: LEADER
I20260925 10:08:04.095364 16138 consensus_queue.cc:237] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } }
I20260925 10:08:04.095722 16173 tablet_replica.cc:439] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24: TxnStatusTablet state changed. Reason: New leader 42ff1943cf664ea682ad77b1709cda24. Latest consensus state: current_term: 1 leader_uuid: "42ff1943cf664ea682ad77b1709cda24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } } }
I20260925 10:08:04.095817 16173 tablet_replica.cc:442] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:08:04.096002 16175 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 10:08:04.096122 16175 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 10:08:04.096230 15619 catalog_manager.cc:5495] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42ff1943cf664ea682ad77b1709cda24 (127.14.41.67). New cstate: current_term: 1 leader_uuid: "42ff1943cf664ea682ad77b1709cda24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } health_report { overall_health: HEALTHY } } }
I20260925 10:08:04.114777 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:04.112226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45122 (local address 127.14.41.67:33801)
0925 10:08:04.112276 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:04.112279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:04.112289 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:08:04.112377 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:04.112378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:04.112398 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:04.112459 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:04.112598 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.113152 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.113935 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.114148 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.114238 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:04.114428 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:04.114535 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:04.114618 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:04.114672 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 10:08:04.114692 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:08:04.114781 16168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:04.112150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:33801 (local address 127.0.0.1:45122)
0925 10:08:04.112230 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:04.112243 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:04.112297 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:04.112468 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:04.112469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:04.112471 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:08:04.112525 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:04.112527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.113247 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.113248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:04.113835 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:04.113836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.114236 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.114237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:04.114307 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:04.114326 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:08:04.114602 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:08:04.114612 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 10:08:04.114667 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 10:08:04.114713 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 10:08:04.116020 15920 raft_consensus.cc:1261] T 086ae995f1d54d17abc00acf61630b7c P 08d887e583964f3e9724a9cf0d5837e4 [term 1 FOLLOWER]: Refusing update from remote peer 42ff1943cf664ea682ad77b1709cda24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:04.116166 15779 raft_consensus.cc:1261] T 086ae995f1d54d17abc00acf61630b7c P 86ce3c8a927340d78c19f0c5e1c386d9 [term 1 FOLLOWER]: Refusing update from remote peer 42ff1943cf664ea682ad77b1709cda24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:04.116444 16138 consensus_queue.cc:1035] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:04.116600 16173 consensus_queue.cc:1035] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:04.117297 16137 tablet_replica.cc:439] T 086ae995f1d54d17abc00acf61630b7c P 08d887e583964f3e9724a9cf0d5837e4: TxnStatusTablet state changed. Reason: New leader 42ff1943cf664ea682ad77b1709cda24. Latest consensus state: current_term: 1 leader_uuid: "42ff1943cf664ea682ad77b1709cda24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } } }
I20260925 10:08:04.117344 16149 tablet_replica.cc:439] T 086ae995f1d54d17abc00acf61630b7c P 86ce3c8a927340d78c19f0c5e1c386d9: TxnStatusTablet state changed. Reason: New leader 42ff1943cf664ea682ad77b1709cda24. Latest consensus state: current_term: 1 leader_uuid: "42ff1943cf664ea682ad77b1709cda24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } } }
I20260925 10:08:04.117394 16137 tablet_replica.cc:442] T 086ae995f1d54d17abc00acf61630b7c P 08d887e583964f3e9724a9cf0d5837e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:08:04.117424 16149 tablet_replica.cc:442] T 086ae995f1d54d17abc00acf61630b7c P 86ce3c8a927340d78c19f0c5e1c386d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:08:04.118188 16173 tablet_replica.cc:439] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "42ff1943cf664ea682ad77b1709cda24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } } }
I20260925 10:08:04.118273 16173 tablet_replica.cc:442] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:08:04.118296 16138 tablet_replica.cc:439] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "42ff1943cf664ea682ad77b1709cda24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } } }
I20260925 10:08:04.118345 16138 tablet_replica.cc:442] T 086ae995f1d54d17abc00acf61630b7c P 42ff1943cf664ea682ad77b1709cda24: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:08:04.118575 16149 tablet_replica.cc:439] T 086ae995f1d54d17abc00acf61630b7c P 86ce3c8a927340d78c19f0c5e1c386d9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "42ff1943cf664ea682ad77b1709cda24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } } }
I20260925 10:08:04.118662 16149 tablet_replica.cc:442] T 086ae995f1d54d17abc00acf61630b7c P 86ce3c8a927340d78c19f0c5e1c386d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:08:04.118824 16137 tablet_replica.cc:439] T 086ae995f1d54d17abc00acf61630b7c P 08d887e583964f3e9724a9cf0d5837e4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "42ff1943cf664ea682ad77b1709cda24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 } } peers { permanent_uuid: "08d887e583964f3e9724a9cf0d5837e4" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45437 } } peers { permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 } } }
I20260925 10:08:04.118916 16137 tablet_replica.cc:442] T 086ae995f1d54d17abc00acf61630b7c P 08d887e583964f3e9724a9cf0d5837e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:08:04.123231 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:04.120726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46560 (local address 127.14.41.66:45437)
0925 10:08:04.120787 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:04.120789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:04.120796 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:08:04.120826 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:04.120826 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:04.120844 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:04.120902 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:04.121043 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.121530 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.122569 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.122744 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.122795 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:04.122944 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:04.123038 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:04.123088 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:04.123157 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 10:08:04.123176 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:08:04.128576 16183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:04.125846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:33801 (local address 127.14.41.66:48919)
0925 10:08:04.126003 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:04.126014 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:04.126073 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:04.126195 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:04.126195 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:04.126196 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:08:04.126248 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:04.126249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.127097 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.127098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:04.127691 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:04.127693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.128200 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.128201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:04.128253 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:04.128267 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:08:04.128438 (+ 171us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:08:04.128439 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 10:08:04.128471 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 10:08:04.128507 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20260925 10:08:04.128572 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:04.125939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:48919 (local address 127.14.41.67:33801)
0925 10:08:04.125993 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:04.125995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:04.126063 (+ 68us) server_negotiation.cc:407] Connection header received
0925 10:08:04.126116 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:04.126117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:04.126127 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:04.126200 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:04.126287 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.126985 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.127778 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.128115 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.128179 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:04.128344 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:04.128379 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:04.128438 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:04.128474 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 10:08:04.128495 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:08:04.130815 15779 raft_consensus.cc:1261] T 2c80ed3bf71d40b18095759cf8b86659 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 1 FOLLOWER]: Refusing update from remote peer 08d887e583964f3e9724a9cf0d5837e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:04.130877 16061 raft_consensus.cc:1261] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24 [term 1 FOLLOWER]: Refusing update from remote peer 08d887e583964f3e9724a9cf0d5837e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:04.131086 16137 consensus_queue.cc:1035] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42ff1943cf664ea682ad77b1709cda24" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:04.131386 16161 consensus_queue.cc:1035] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86ce3c8a927340d78c19f0c5e1c386d9" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 46701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:04.140348 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:04.137289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:45437 (local address 127.14.41.67:39081)
0925 10:08:04.137372 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:04.137385 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:04.137430 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:04.137615 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:04.137616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:04.137617 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:08:04.137691 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:04.137693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.138600 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.138601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:04.139323 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:04.139326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.139831 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.139832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:04.139902 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:04.139922 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:08:04.140137 (+ 215us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:08:04.140140 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:04.140201 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 10:08:04.140250 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 10:08:04.140663 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:04.137406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:39081 (local address 127.14.41.66:45437)
0925 10:08:04.137445 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:04.137447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:04.137455 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:08:04.137479 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:04.137480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:04.137489 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:04.137536 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:04.137773 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.138485 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.139412 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.139735 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.139828 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:04.140024 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:04.140064 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:04.140111 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:04.140579 (+ 468us) server_negotiation.cc:299] Negotiation successful
0925 10:08:04.140604 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:08:04.150496 15618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:04.150624 15618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:04.151331 15618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6352a46aef9b41ac978565508dc8cc0e: Sending DeleteTablet for 3 replicas of tablet 2c80ed3bf71d40b18095759cf8b86659
I20260925 10:08:04.151603 15759 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c80ed3bf71d40b18095759cf8b86659 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:04 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:50734
I20260925 10:08:04.151775 16156 tablet_replica.cc:330] T 2c80ed3bf71d40b18095759cf8b86659 P 86ce3c8a927340d78c19f0c5e1c386d9: stopping tablet replica
I20260925 10:08:04.151873 16156 raft_consensus.cc:2229] T 2c80ed3bf71d40b18095759cf8b86659 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:04.151952 16156 raft_consensus.cc:2258] T 2c80ed3bf71d40b18095759cf8b86659 P 86ce3c8a927340d78c19f0c5e1c386d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:04.152107 15900 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c80ed3bf71d40b18095759cf8b86659 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:04 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:46554
I20260925 10:08:04.152204 16155 tablet_replica.cc:330] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4: stopping tablet replica
I20260925 10:08:04.152259 16155 raft_consensus.cc:2229] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:08:04.152359 16155 raft_consensus.cc:2258] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:04.152740 16155 ts_tablet_manager.cc:1905] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:04.152810 16041 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c80ed3bf71d40b18095759cf8b86659 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:04 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:45120
I20260925 10:08:04.152917 16154 tablet_replica.cc:330] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24: stopping tablet replica
I20260925 10:08:04.152968 16154 raft_consensus.cc:2229] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:04.153005 16154 raft_consensus.cc:2258] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:04.153120 16156 ts_tablet_manager.cc:1905] T 2c80ed3bf71d40b18095759cf8b86659 P 86ce3c8a927340d78c19f0c5e1c386d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:04.154660 16154 ts_tablet_manager.cc:1905] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:04.154969 16156 ts_tablet_manager.cc:1918] T 2c80ed3bf71d40b18095759cf8b86659 P 86ce3c8a927340d78c19f0c5e1c386d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:08:04.155054 16156 log.cc:1196] T 2c80ed3bf71d40b18095759cf8b86659 P 86ce3c8a927340d78c19f0c5e1c386d9: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-0/wal/wals/2c80ed3bf71d40b18095759cf8b86659
I20260925 10:08:04.155401 16156 ts_tablet_manager.cc:1939] T 2c80ed3bf71d40b18095759cf8b86659 P 86ce3c8a927340d78c19f0c5e1c386d9: Deleting consensus metadata
I20260925 10:08:04.156127 15604 catalog_manager.cc:4841] TS 86ce3c8a927340d78c19f0c5e1c386d9 (127.14.41.65:46701): tablet 2c80ed3bf71d40b18095759cf8b86659 (table test-table [id=e86682cbc11447df86ef180a9cf8f677]) successfully deleted
I20260925 10:08:04.157330 16154 ts_tablet_manager.cc:1918] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:08:04.157401 16154 log.cc:1196] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/2c80ed3bf71d40b18095759cf8b86659
I20260925 10:08:04.157567 16155 ts_tablet_manager.cc:1918] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:08:04.157626 16155 log.cc:1196] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/2c80ed3bf71d40b18095759cf8b86659
I20260925 10:08:04.157629 16154 ts_tablet_manager.cc:1939] T 2c80ed3bf71d40b18095759cf8b86659 P 42ff1943cf664ea682ad77b1709cda24: Deleting consensus metadata
I20260925 10:08:04.157840 16155 ts_tablet_manager.cc:1939] T 2c80ed3bf71d40b18095759cf8b86659 P 08d887e583964f3e9724a9cf0d5837e4: Deleting consensus metadata
I20260925 10:08:04.157982 15604 catalog_manager.cc:4841] TS 42ff1943cf664ea682ad77b1709cda24 (127.14.41.67:33801): tablet 2c80ed3bf71d40b18095759cf8b86659 (table test-table [id=e86682cbc11447df86ef180a9cf8f677]) successfully deleted
I20260925 10:08:04.158349 15604 catalog_manager.cc:4841] TS 08d887e583964f3e9724a9cf0d5837e4 (127.14.41.66:45437): tablet 2c80ed3bf71d40b18095759cf8b86659 (table test-table [id=e86682cbc11447df86ef180a9cf8f677]) successfully deleted
Sep 25 10:08:04 dist-test-slave-s6tm krb5kdc[15574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330883, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.41.65@KRBTEST.COM
I20260925 10:08:04.169261 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:04.156217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50754 (local address 127.14.41.65:46701)
0925 10:08:04.156263 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:04.156265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:04.158380 (+ 2115us) server_negotiation.cc:407] Connection header received
0925 10:08:04.158558 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:04.158559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:04.158572 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:04.158638 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:04.159391 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.160176 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.161259 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.161600 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.161681 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:04.165470 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:04.165487 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:04.165487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:04.165497 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:04.167829 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:04.168335 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:04.168339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:04.168340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:04.168411 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:04.168635 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:04.168635 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:04.168636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:04.168835 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:04.168979 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:04.169062 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 10:08:04.169155 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 10:08:04.169510 15819 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50754
I20260925 10:08:04.180223 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:04.172236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54106 (local address 127.14.41.126:42637)
0925 10:08:04.172281 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:04.172283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:04.172399 (+ 116us) server_negotiation.cc:407] Connection header received
0925 10:08:04.172500 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:04.172501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:04.172513 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:04.172543 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:04.173271 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.174087 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.175136 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:04.175496 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:04.175540 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:04.177173 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:04.177175 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:04.177175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:04.177183 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:04.178929 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:04.179334 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:04.179336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:04.179336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:04.179392 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:04.179591 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:04.179592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:04.179592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:04.179717 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:04.179788 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:04.180034 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 10:08:04.180115 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 10:08:04.180852 15618 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54106
I20260925 10:08:04.181609 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 15693
I20260925 10:08:04.190572 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 15834
I20260925 10:08:04.196192 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 15975
I20260925 10:08:04.201601 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 15587
2026-09-25T10:08:04Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2411 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:08:04 dist-test-slave-s6tm krb5kdc[16196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:08:04 dist-test-slave-s6tm krb5kdc[16196](info): set up 2 sockets
Sep 25 10:08:04 dist-test-slave-s6tm krb5kdc[16196](info): commencing operation
krb5kdc: starting...
W20260925 10:08:05.273414 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:08:05 dist-test-slave-s6tm krb5kdc[16196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:08:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:08:05.400811 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:37185
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:37185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:05.487536 16210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:05.487803 16210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:05.487843 16210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:05.489106 16210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:05.489182 16210 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 10:08:05.489204 16210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:05.489228 16210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:05.489254 16210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:05.489287 16210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:08:05.490861 16210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:37185
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:37185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:05.491171 16210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:05.491420 16210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:05.497119 16218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:05.497218 16215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:05.497131 16216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:05.497368 16210 server_base.cc:1034] running on GCE node
I20260925 10:08:05.497614 16210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:05.497884 16210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:05.499085 16210 hybrid_clock.cc:648] HybridClock initialized: now 1790330885499053 us; error 48 us; skew 500 ppm
Sep 25 10:08:05 dist-test-slave-s6tm krb5kdc[16196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330885, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:05.502004 16210 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:08:05.502489 16210 webserver.cc:460] Webserver started at http://127.14.41.126:38731/ using document root <none> and password file <none>
I20260925 10:08:05.502748 16210 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:05.502827 16210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:05.502992 16210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:05.504271 16210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "9a6f47c71e4d4988ab90685daa68c526"
format_stamp: "Formatted at 2026-09-25 10:08:05 on dist-test-slave-s6tm"
server_key: "6c0f9f992b80af89919bf10cc1383550"
server_key_iv: "87359fd798be778090b3a3057bbe0e50"
server_key_version: "encryptionkey@0"
I20260925 10:08:05.504673 16210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "9a6f47c71e4d4988ab90685daa68c526"
format_stamp: "Formatted at 2026-09-25 10:08:05 on dist-test-slave-s6tm"
server_key: "6c0f9f992b80af89919bf10cc1383550"
server_key_iv: "87359fd798be778090b3a3057bbe0e50"
server_key_version: "encryptionkey@0"
I20260925 10:08:05.506247 16210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:08:05.507138 16225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:05.507351 16210 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:05.507469 16210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "9a6f47c71e4d4988ab90685daa68c526"
format_stamp: "Formatted at 2026-09-25 10:08:05 on dist-test-slave-s6tm"
server_key: "6c0f9f992b80af89919bf10cc1383550"
server_key_iv: "87359fd798be778090b3a3057bbe0e50"
server_key_version: "encryptionkey@0"
I20260925 10:08:05.507550 16210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:05.517621 16210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:05.519675 16210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:05.519850 16210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:05.524718 16210 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:37185
I20260925 10:08:05.524802 16287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:37185 every 8 connection(s)
I20260925 10:08:05.525147 16210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:05.525847 16288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:05.526360 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 16210
I20260925 10:08:05.526453 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:08:05.526616 14501 external_mini_cluster.cc:1428] Setting key 4625b5b301aa85a3bbb1db26eb121f7a
I20260925 10:08:05.529615 16288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526: Bootstrap starting.
I20260925 10:08:05.530344 16288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:05.530648 16288 log.cc:824] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:05.531461 16288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526: No bootstrap required, opened a new log
Sep 25 10:08:05 dist-test-slave-s6tm krb5kdc[16196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:05.534408 16288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6f47c71e4d4988ab90685daa68c526" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37185 } }
I20260925 10:08:05.534571 16288 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:05.534598 16288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a6f47c71e4d4988ab90685daa68c526, State: Initialized, Role: FOLLOWER
I20260925 10:08:05.534770 16288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6f47c71e4d4988ab90685daa68c526" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37185 } }
I20260925 10:08:05.534921 16288 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:08:05.534976 16288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:05.535056 16288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:05.535684 16288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6f47c71e4d4988ab90685daa68c526" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37185 } }
I20260925 10:08:05.535815 16288 leader_election.cc:302] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a6f47c71e4d4988ab90685daa68c526; no voters:
I20260925 10:08:05.536068 16288 leader_election.cc:288] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:08:05.536137 16293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:05.536298 16293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526 [term 1 LEADER]: Becoming Leader. State: Replica: 9a6f47c71e4d4988ab90685daa68c526, State: Running, Role: LEADER
I20260925 10:08:05.536391 16288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:05.536442 16293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6f47c71e4d4988ab90685daa68c526" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37185 } }
I20260925 10:08:05.537091 16294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a6f47c71e4d4988ab90685daa68c526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6f47c71e4d4988ab90685daa68c526" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37185 } } }
I20260925 10:08:05.537189 16294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:05.537439 16297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:05.538034 16297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:05.539414 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:05.527146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59970 (local address 127.14.41.126:37185)
0925 10:08:05.527301 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:05.527312 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:05.527346 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:08:05.527844 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:05.527853 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:05.527953 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:05.528108 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:05.528740 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:05.530139 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:05.531105 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:05.531640 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:05.531724 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:05.534736 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:05.534740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:05.534741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:05.534755 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:05.537265 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:05.537708 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:05.537711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:05.537712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:05.537822 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:05.538051 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:05.538052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:05.538053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:05.538360 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:05.538512 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:05.539072 (+ 560us) server_negotiation.cc:299] Negotiation successful
0925 10:08:05.539217 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260925 10:08:05.539518 16295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a6f47c71e4d4988ab90685daa68c526. Latest consensus state: current_term: 1 leader_uuid: "9a6f47c71e4d4988ab90685daa68c526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6f47c71e4d4988ab90685daa68c526" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37185 } } }
I20260925 10:08:05.539589 16295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:05.539783 16297 catalog_manager.cc:1347] Generated new cluster ID: 6d9e8064ce234d839903323a110c69db
I20260925 10:08:05.539824 16297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:08:05.547448 16297 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:08:05.548171 16297 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:05.550356 16297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526: Generated new TSK 0
I20260925 10:08:05.550556 16297 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
I20260925 10:08:05.587412 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37185
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:05.690703 16314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:05.690979 16314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:05.691030 16314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:05.692447 16314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:05.692521 16314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:05.692561 16314 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 10:08:05.692615 16314 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:08:05.692658 16314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:08:05.694795 16314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.41.126:37185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:05.695142 16314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:05.695405 16314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:05.700567 16322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:05.700563 16320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:05.700578 16319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:05.701159 16314 server_base.cc:1034] running on GCE node
I20260925 10:08:05.701323 16314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:05.701524 16314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:05.702674 16314 hybrid_clock.cc:648] HybridClock initialized: now 1790330885702660 us; error 29 us; skew 500 ppm
Sep 25 10:08:05 dist-test-slave-s6tm krb5kdc[16196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330885, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:05.705346 16314 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:08:05.705737 16314 webserver.cc:460] Webserver started at http://127.14.41.65:41483/ using document root <none> and password file <none>
I20260925 10:08:05.705998 16314 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:05.706071 16314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:05.706226 16314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:05.707309 16314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "63f3197999764c8cadbdd94ede65a75f"
format_stamp: "Formatted at 2026-09-25 10:08:05 on dist-test-slave-s6tm"
server_key: "87a35e1fe6acd1550052e74de7c17cb6"
server_key_iv: "5df5d3528b64fccb6fdd174804da6e52"
server_key_version: "encryptionkey@0"
I20260925 10:08:05.707686 16314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "63f3197999764c8cadbdd94ede65a75f"
format_stamp: "Formatted at 2026-09-25 10:08:05 on dist-test-slave-s6tm"
server_key: "87a35e1fe6acd1550052e74de7c17cb6"
server_key_iv: "5df5d3528b64fccb6fdd174804da6e52"
server_key_version: "encryptionkey@0"
I20260925 10:08:05.709172 16314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:08:05.709991 16329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:05.710203 16314 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:05.710305 16314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "63f3197999764c8cadbdd94ede65a75f"
format_stamp: "Formatted at 2026-09-25 10:08:05 on dist-test-slave-s6tm"
server_key: "87a35e1fe6acd1550052e74de7c17cb6"
server_key_iv: "5df5d3528b64fccb6fdd174804da6e52"
server_key_version: "encryptionkey@0"
I20260925 10:08:05.710403 16314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:05.721590 16314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:05.724212 16314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:05.724380 16314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:05.724923 16314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:05.724989 16314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:05.725037 16314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:05.725064 16314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:05.731767 16314 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:34163
I20260925 10:08:05.731979 16445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:34163 every 8 connection(s)
I20260925 10:08:05.732235 16314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
Sep 25 10:08:05 dist-test-slave-s6tm krb5kdc[16196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330885, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:05.733801 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 16314
I20260925 10:08:05.733898 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:08:05.734109 14501 external_mini_cluster.cc:1428] Setting key ad897435cc86fb7f2a78cd67cdeb569c
I20260925 10:08:05.737273 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:05.725483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:37185 (local address 127.14.41.65:57881)
0925 10:08:05.725657 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:05.725702 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:05.726248 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:05.726599 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:05.726604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:05.727008 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:05.727366 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:05.727375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:05.728640 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:05.728641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:05.730106 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:05.730109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:05.730701 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:05.730702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:05.730886 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:05.731522 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:05.731533 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:05.733473 (+ 1940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:05.735158 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:05.735165 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:05.735167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:05.735528 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:05.735778 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:05.735779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:05.735779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:05.735962 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:05.736225 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:05.736232 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:05.736456 (+ 224us) client_negotiation.cc:769] Sending connection context
0925 10:08:05.736624 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 10:08:05.737086 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260925 10:08:05.737726 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:05.725500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:57881 (local address 127.14.41.126:37185)
0925 10:08:05.725564 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:05.725568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:05.725799 (+ 231us) server_negotiation.cc:407] Connection header received
0925 10:08:05.726394 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:05.726396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:05.726419 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:05.726501 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:05.727493 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:05.728513 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:05.730201 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:05.730584 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:05.730664 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:05.733569 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:05.733571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:05.733571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:05.733579 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:05.735078 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:05.735608 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:05.735609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:05.735610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:05.735661 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:05.736028 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:05.736028 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:05.736029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:05.736157 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:05.736214 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:05.737505 (+ 1291us) server_negotiation.cc:299] Negotiation successful
0925 10:08:05.737617 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 10:08:05.738231 16446 heartbeater.cc:347] Connected to a master server at 127.14.41.126:37185
I20260925 10:08:05.738386 16446 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:05.738642 16446 heartbeater.cc:511] Master 127.14.41.126:37185 requested a full tablet report, sending...
I20260925 10:08:05.739375 16242 ts_manager.cc:194] Registered new tserver with Master: 63f3197999764c8cadbdd94ede65a75f (127.14.41.65:34163)
I20260925 10:08:05.740520 16242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:57881
WARNING: no policy specified for kudu/127.14.41.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.66@KRBTEST.COM" created.
I20260925 10:08:05.754698 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:05.739654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:35637 (local address 127.14.41.126:37185)
0925 10:08:05.739694 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:05.739696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:05.739892 (+ 196us) server_negotiation.cc:407] Connection header received
0925 10:08:05.740021 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:05.740023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:05.740033 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:05.740063 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:05.740674 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:05.741385 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:05.742230 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:05.742443 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:05.742531 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:05.743617 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:05.743619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:05.743619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:05.743626 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:05.753361 (+ 9735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:05.753809 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:05.753811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:05.753811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:05.753877 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:05.754091 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:05.754092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:05.754092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:05.754220 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:05.754301 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:05.754504 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 10:08:05.754593 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:08:05.754702 16453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:05.739585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:37185 (local address 127.14.41.65:35637)
0925 10:08:05.739764 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:05.739776 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:05.739920 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:05.740124 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:05.740125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:05.740490 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:05.740564 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:05.740567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:05.741504 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:05.741505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:05.742116 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:05.742119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:05.742531 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:05.742531 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:05.742606 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:05.743102 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:05.743107 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:05.743506 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:05.753505 (+ 9999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:05.753507 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:05.753507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:05.753717 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:05.753958 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:05.753959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:05.753959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:05.754026 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:05.754327 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:05.754328 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:05.754391 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 10:08:05.754469 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 10:08:05.754593 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":73,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
I20260925 10:08:05.763309 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37185
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:05.854218 16455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:05.854449 16455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:05.854477 16455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:05.855808 16455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:05.855860 16455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:05.855885 16455 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 10:08:05.855898 16455 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:08:05.856001 16455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:08:05.857630 16455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.41.126:37185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:05.857920 16455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:05.858170 16455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:05.863787 16463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:05.863816 16460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:05.863791 16455 server_base.cc:1034] running on GCE node
W20260925 10:08:05.864156 16461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:05.864374 16455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:05.864637 16455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:05.865803 16455 hybrid_clock.cc:648] HybridClock initialized: now 1790330885865790 us; error 27 us; skew 500 ppm
Sep 25 10:08:05 dist-test-slave-s6tm krb5kdc[16196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330885, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:05.869334 16455 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:08:05.870038 16455 webserver.cc:460] Webserver started at http://127.14.41.66:45785/ using document root <none> and password file <none>
I20260925 10:08:05.870339 16455 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:05.870419 16455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:05.870612 16455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:05.871816 16455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "d114b21c51064f0292a65a3aa3c55b34"
format_stamp: "Formatted at 2026-09-25 10:08:05 on dist-test-slave-s6tm"
server_key: "d4848c596d31a4e32ce006ca517a2604"
server_key_iv: "6c0770c57d26a4765f1684bd8a4be3a1"
server_key_version: "encryptionkey@0"
I20260925 10:08:05.872337 16455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "d114b21c51064f0292a65a3aa3c55b34"
format_stamp: "Formatted at 2026-09-25 10:08:05 on dist-test-slave-s6tm"
server_key: "d4848c596d31a4e32ce006ca517a2604"
server_key_iv: "6c0770c57d26a4765f1684bd8a4be3a1"
server_key_version: "encryptionkey@0"
I20260925 10:08:05.873968 16455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:08:05.874920 16470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:05.875197 16455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:05.875327 16455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "d114b21c51064f0292a65a3aa3c55b34"
format_stamp: "Formatted at 2026-09-25 10:08:05 on dist-test-slave-s6tm"
server_key: "d4848c596d31a4e32ce006ca517a2604"
server_key_iv: "6c0770c57d26a4765f1684bd8a4be3a1"
server_key_version: "encryptionkey@0"
I20260925 10:08:05.875419 16455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:05.886818 16455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:05.889251 16455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:05.889439 16455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:05.890086 16455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:05.890157 16455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:05.890206 16455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:05.890244 16455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:08:05 dist-test-slave-s6tm krb5kdc[16196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330885, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:05.898370 16455 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:44873
I20260925 10:08:05.898540 16585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:44873 every 8 connection(s)
I20260925 10:08:05.898829 16455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:08:05.899147 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 16455
I20260925 10:08:05.899250 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:08:05.899456 14501 external_mini_cluster.cc:1428] Setting key feaea673471b8ec906ca2ce07b500c2e
I20260925 10:08:05.901492 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:05.890592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:37185 (local address 127.14.41.66:47383)
0925 10:08:05.890739 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:05.890780 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:05.891309 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:05.891583 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:05.891586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:05.891973 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:05.892228 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:05.892233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:05.893001 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:05.893002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:05.893923 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:05.893925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:05.894380 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:05.894381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:05.894537 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:05.895118 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:05.895130 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:05.897886 (+ 2756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:05.900053 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:05.900059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:05.900061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:05.900369 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:05.900586 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:05.900586 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:05.900587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:05.900683 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:05.900989 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:05.900994 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:05.901109 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 10:08:05.901198 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 10:08:05.901330 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":91,"threads_started":1}
I20260925 10:08:05.901510 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:05.890578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:47383 (local address 127.14.41.126:37185)
0925 10:08:05.890621 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:05.890627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:05.890867 (+ 240us) server_negotiation.cc:407] Connection header received
0925 10:08:05.891422 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:05.891424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:05.891438 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:05.891504 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:05.892341 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:05.892896 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:05.894026 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:05.894278 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:05.894357 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:05.897999 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:05.898002 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:05.898002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:05.898011 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:05.899980 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:05.900447 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:05.900448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:05.900448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:05.900518 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:05.900747 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:05.900748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:05.900749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:05.900917 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:05.901003 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:05.901251 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 10:08:05.901351 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:08:05.902100 16586 heartbeater.cc:347] Connected to a master server at 127.14.41.126:37185
I20260925 10:08:05.902210 16586 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:05.902428 16586 heartbeater.cc:511] Master 127.14.41.126:37185 requested a full tablet report, sending...
I20260925 10:08:05.902815 16241 ts_manager.cc:194] Registered new tserver with Master: d114b21c51064f0292a65a3aa3c55b34 (127.14.41.66:44873)
I20260925 10:08:05.903419 16241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:47383
WARNING: no policy specified for kudu/127.14.41.67@KRBTEST.COM; defaulting to no policy
I20260925 10:08:05.910898 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:05.903165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:41435 (local address 127.14.41.126:37185)
0925 10:08:05.903205 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:05.903207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:05.903778 (+ 571us) server_negotiation.cc:407] Connection header received
0925 10:08:05.903835 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:05.903836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:05.903846 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:05.903881 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:05.904985 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:05.905466 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:05.906752 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:05.906973 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:05.907029 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:05.908150 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:05.908151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:05.908151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:05.908158 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:05.909652 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:05.910051 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:05.910052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:05.910053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:05.910110 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:05.910354 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:05.910355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:05.910355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:05.910520 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:05.910574 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:05.910707 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 10:08:05.910790 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:08:05.910919 16593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:05.903126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:37185 (local address 127.14.41.66:41435)
0925 10:08:05.903698 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:05.903711 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:05.903773 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:05.904547 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:05.904548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:05.904791 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:05.904871 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:05.904873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:05.905583 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:05.905584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:05.906668 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:05.906671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:05.907057 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:05.907059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:05.907145 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:05.907578 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:05.907584 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:05.908057 (+ 473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:05.909766 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:05.909768 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:05.909768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:05.909962 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:05.910180 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:05.910180 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:05.910181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:05.910283 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:05.910567 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:05.910567 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:05.910608 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 10:08:05.910690 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 10:08:05.910801 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":66,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
I20260925 10:08:05.928639 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37185
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:06.017997 16595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:06.018242 16595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:06.018272 16595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:06.019523 16595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:06.019568 16595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:06.019591 16595 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 10:08:06.019604 16595 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:08:06.019623 16595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:08:06.021312 16595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.41.126:37185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:06.021546 16595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:06.021757 16595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:06.027554 16600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:06.027565 16601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:06.027666 16595 server_base.cc:1034] running on GCE node
W20260925 10:08:06.027632 16603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:06.028062 16595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:06.028358 16595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:06.029536 16595 hybrid_clock.cc:648] HybridClock initialized: now 1790330886029525 us; error 41 us; skew 500 ppm
Sep 25 10:08:06 dist-test-slave-s6tm krb5kdc[16196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330886, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:06.032286 16595 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:08:06.032769 16595 webserver.cc:460] Webserver started at http://127.14.41.67:34139/ using document root <none> and password file <none>
I20260925 10:08:06.033041 16595 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:06.033113 16595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:06.033272 16595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:06.034384 16595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "af3a8c6420224a1e98f2e272f0a0aa4c"
format_stamp: "Formatted at 2026-09-25 10:08:06 on dist-test-slave-s6tm"
server_key: "f176b36fe25aa7ae9c241f62dc173b00"
server_key_iv: "0abae123d64b3dc65e73fe09ab823c01"
server_key_version: "encryptionkey@0"
I20260925 10:08:06.034757 16595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "af3a8c6420224a1e98f2e272f0a0aa4c"
format_stamp: "Formatted at 2026-09-25 10:08:06 on dist-test-slave-s6tm"
server_key: "f176b36fe25aa7ae9c241f62dc173b00"
server_key_iv: "0abae123d64b3dc65e73fe09ab823c01"
server_key_version: "encryptionkey@0"
I20260925 10:08:06.036394 16595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260925 10:08:06.037319 16610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:06.037590 16595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:06.037688 16595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "af3a8c6420224a1e98f2e272f0a0aa4c"
format_stamp: "Formatted at 2026-09-25 10:08:06 on dist-test-slave-s6tm"
server_key: "f176b36fe25aa7ae9c241f62dc173b00"
server_key_iv: "0abae123d64b3dc65e73fe09ab823c01"
server_key_version: "encryptionkey@0"
I20260925 10:08:06.037793 16595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:06.053061 16595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:06.055539 16595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:06.055722 16595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:06.056411 16595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:06.056490 16595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:06.056530 16595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:06.056557 16595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:08:06 dist-test-slave-s6tm krb5kdc[16196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330886, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:06.064416 16595 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:34993
I20260925 10:08:06.064510 16726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:34993 every 8 connection(s)
I20260925 10:08:06.064857 16595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:08:06.067413 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.056949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:44837 (local address 127.14.41.126:37185)
0925 10:08:06.057022 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:06.057025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:06.057256 (+ 231us) server_negotiation.cc:407] Connection header received
0925 10:08:06.057781 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:06.057783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:06.057797 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:06.057872 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:06.058843 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.059698 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.061030 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.061239 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.061373 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.064398 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:06.064401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:06.064401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:06.064412 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:06.066105 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:06.066534 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:06.066535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:06.066536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:06.066594 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:06.066823 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:06.066823 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:06.066824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:06.066958 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:06.067013 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:06.067239 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 10:08:06.067346 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:08:06.067714 16619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.056929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:37185 (local address 127.14.41.67:44837)
0925 10:08:06.057107 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:06.057149 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:06.057665 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:06.057985 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:06.057989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:06.058365 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:06.058708 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.058715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.059832 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.059833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.060895 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.060898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.061406 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.061407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.061543 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.062198 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:06.062210 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:06.064274 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:06.066200 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:06.066207 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:06.066209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:06.066455 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:06.066656 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:06.066657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:06.066657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:06.066758 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:06.067015 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:06.067020 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:06.067144 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 10:08:06.067229 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 10:08:06.067364 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
I20260925 10:08:06.068279 16727 heartbeater.cc:347] Connected to a master server at 127.14.41.126:37185
I20260925 10:08:06.068384 16727 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:06.068583 16727 heartbeater.cc:511] Master 127.14.41.126:37185 requested a full tablet report, sending...
I20260925 10:08:06.068980 16242 ts_manager.cc:194] Registered new tserver with Master: af3a8c6420224a1e98f2e272f0a0aa4c (127.14.41.67:34993)
I20260925 10:08:06.069501 16242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:44837
I20260925 10:08:06.074262 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 16595
I20260925 10:08:06.074424 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:08:06.074644 14501 external_mini_cluster.cc:1428] Setting key db5c9945c8708d84b60e3548f63d112a
I20260925 10:08:06.075311 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.069328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:50183 (local address 127.14.41.126:37185)
0925 10:08:06.069372 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:06.069374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:06.069470 (+ 96us) server_negotiation.cc:407] Connection header received
0925 10:08:06.069534 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:06.069535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:06.069546 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:06.069595 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:06.069996 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.070485 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.071202 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.071417 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.071497 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.072594 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:06.072596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:06.072596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:06.072604 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:06.073928 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:06.074393 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:06.074394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:06.074395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:06.074514 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:06.074740 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:06.074741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:06.074742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:06.074870 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:06.074953 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:06.075116 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 10:08:06.075201 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:08:06.075323 16733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.069270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:37185 (local address 127.14.41.67:50183)
0925 10:08:06.069389 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:06.069400 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:06.069472 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:06.069597 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:06.069597 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:06.069833 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:06.069898 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.069900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.070588 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.070589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.071112 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.071114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.071495 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.071496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.071573 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.072173 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:06.072178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:06.072490 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:06.074048 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:06.074049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:06.074050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:06.074287 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:06.074598 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:06.074599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:06.074599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:06.074670 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:06.074943 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:06.074944 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:06.074995 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 10:08:06.075086 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 10:08:06.075204 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
I20260925 10:08:06.075644 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:08:06 dist-test-slave-s6tm krb5kdc[16196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:08:06 dist-test-slave-s6tm krb5kdc[16196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:06.100656 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.092714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59978 (local address 127.14.41.126:37185)
0925 10:08:06.092797 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:06.092800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:06.092821 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:08:06.092856 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:06.092857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:06.092865 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:06.092940 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:06.093671 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.094188 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.094980 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.095177 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.095269 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.097807 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:06.097809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:06.097809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:06.097816 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:06.099179 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:06.099619 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:06.099621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:06.099621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:06.099683 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:06.099990 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:06.099991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:06.099992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:06.100144 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:06.100257 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:06.100447 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 10:08:06.100547 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:08:06.102504 16242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:08:06.104516 16242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:08:06.111814 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.107276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50486 (local address 127.14.41.66:44873)
0925 10:08:06.107609 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:06.107614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:06.107624 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:08:06.107774 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:06.107776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:06.107794 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:06.107876 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:06.108162 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.108882 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.110691 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.111345 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.111441 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.111509 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:06.111671 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 10:08:06.111709 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":90,"threads_started":1}
I20260925 10:08:06.111814 16744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.107148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:44873 (local address 127.0.0.1:50486)
0925 10:08:06.107501 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:06.107511 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:06.107567 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:06.107982 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:06.107986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:06.107988 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:06.108064 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.108067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.108983 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.108984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.110578 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.110581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.111445 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.111446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.111528 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.111542 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 10:08:06.111650 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:08:06.111708 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":79,"threads_started":1}
I20260925 10:08:06.112088 16745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.107140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:34163 (local address 127.0.0.1:55622)
0925 10:08:06.107376 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:06.107423 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:06.107496 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:06.108023 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:06.108023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:06.108026 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:06.108110 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.108112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.109538 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.109540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.110756 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.110759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.111810 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.111811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.111877 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.111882 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:08:06.111914 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 10:08:06.112006 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":18,"thread_start_us":59,"threads_started":1}
I20260925 10:08:06.112404 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.107342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:34993 (local address 127.0.0.1:50456)
0925 10:08:06.107642 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:06.107652 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:06.107692 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:06.108246 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:06.108247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:06.108249 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:06.108285 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.108286 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.109835 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.109836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.110927 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.110929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.112206 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.112207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.112267 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.112271 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:08:06.112301 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 10:08:06.112339 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":107,"threads_started":1}
I20260925 10:08:06.112550 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.107268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55622 (local address 127.14.41.65:34163)
0925 10:08:06.107803 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:06.107810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:06.107821 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:08:06.107861 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:06.107864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:06.107880 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:06.108021 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:06.108210 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.109428 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.110859 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.111708 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.112259 (+ 551us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.112332 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:06.112358 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 10:08:06.112398 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":85,"threads_started":1}
I20260925 10:08:06.112802 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.107590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50456 (local address 127.14.41.67:34993)
0925 10:08:06.108077 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:06.108083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:06.108094 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:08:06.108130 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:06.108132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:06.108148 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:06.108352 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:06.108406 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.109430 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.111017 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.111708 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.112573 (+ 865us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.112643 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:06.112668 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 10:08:06.112715 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":97,"threads_started":1}
I20260925 10:08:06.112776 16520 tablet_service.cc:1467] Processing CreateTablet for tablet 847fcc742a9049a094ac6961f7d7037e (DEFAULT_TABLE table=test-table [id=6ecfd6da7aac493185d8273d5fed836f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:06.113166 16520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 847fcc742a9049a094ac6961f7d7037e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:06.113425 16661 tablet_service.cc:1467] Processing CreateTablet for tablet 847fcc742a9049a094ac6961f7d7037e (DEFAULT_TABLE table=test-table [id=6ecfd6da7aac493185d8273d5fed836f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:06.113718 16661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 847fcc742a9049a094ac6961f7d7037e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:06.116449 16750 tablet_bootstrap.cc:492] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34: Bootstrap starting.
I20260925 10:08:06.117056 16750 tablet_bootstrap.cc:654] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:06.117094 16380 tablet_service.cc:1467] Processing CreateTablet for tablet 847fcc742a9049a094ac6961f7d7037e (DEFAULT_TABLE table=test-table [id=6ecfd6da7aac493185d8273d5fed836f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:06.117417 16750 log.cc:824] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:06.117450 16380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 847fcc742a9049a094ac6961f7d7037e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:06.117775 16751 tablet_bootstrap.cc:492] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c: Bootstrap starting.
I20260925 10:08:06.118131 16750 tablet_bootstrap.cc:492] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34: No bootstrap required, opened a new log
I20260925 10:08:06.118186 16750 ts_tablet_manager.cc:1397] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:08:06.119704 16750 raft_consensus.cc:348] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.119863 16750 raft_consensus.cc:374] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:06.119911 16750 raft_consensus.cc:729] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d114b21c51064f0292a65a3aa3c55b34, State: Initialized, Role: FOLLOWER
I20260925 10:08:06.120080 16750 consensus_queue.cc:260] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.120390 16750 ts_tablet_manager.cc:1428] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:08:06.120663 16754 tablet_bootstrap.cc:492] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f: Bootstrap starting.
I20260925 10:08:06.120697 16586 heartbeater.cc:503] Master 127.14.41.126:37185 was elected leader, sending a full tablet report...
I20260925 10:08:06.120903 16751 tablet_bootstrap.cc:654] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:06.121183 16751 log.cc:824] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:06.121321 16754 tablet_bootstrap.cc:654] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:06.121634 16754 log.cc:824] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:06.121973 16751 tablet_bootstrap.cc:492] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c: No bootstrap required, opened a new log
I20260925 10:08:06.122028 16751 ts_tablet_manager.cc:1397] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 10:08:06.122314 16754 tablet_bootstrap.cc:492] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f: No bootstrap required, opened a new log
I20260925 10:08:06.122368 16754 ts_tablet_manager.cc:1397] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:08:06.123780 16751 raft_consensus.cc:348] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.123977 16751 raft_consensus.cc:374] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:06.124019 16751 raft_consensus.cc:729] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3a8c6420224a1e98f2e272f0a0aa4c, State: Initialized, Role: FOLLOWER
I20260925 10:08:06.124137 16751 consensus_queue.cc:260] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.124140 16754 raft_consensus.cc:348] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.124293 16754 raft_consensus.cc:374] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:06.124336 16754 raft_consensus.cc:729] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63f3197999764c8cadbdd94ede65a75f, State: Initialized, Role: FOLLOWER
I20260925 10:08:06.124432 16751 ts_tablet_manager.cc:1428] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:08:06.124451 16727 heartbeater.cc:503] Master 127.14.41.126:37185 was elected leader, sending a full tablet report...
I20260925 10:08:06.124465 16754 consensus_queue.cc:260] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.124714 16754 ts_tablet_manager.cc:1428] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:08:06.124766 16446 heartbeater.cc:503] Master 127.14.41.126:37185 was elected leader, sending a full tablet report...
W20260925 10:08:06.149685 16587 tablet.cc:2369] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:08:06.233101 16447 tablet.cc:2369] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:08:06.243147 16758 raft_consensus.cc:482] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:06.243326 16758 raft_consensus.cc:504] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.243785 16758 leader_election.cc:288] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d114b21c51064f0292a65a3aa3c55b34 (127.14.41.66:44873), af3a8c6420224a1e98f2e272f0a0aa4c (127.14.41.67:34993)
I20260925 10:08:06.246850 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.244165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:36441 (local address 127.14.41.67:34993)
0925 10:08:06.244243 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:06.244245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:06.244291 (+ 46us) server_negotiation.cc:407] Connection header received
0925 10:08:06.244401 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:06.244402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:06.244412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:06.244470 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:06.244641 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.245126 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.246114 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.246491 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.246548 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.246556 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:06.246726 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 10:08:06.246774 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:08:06.246892 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.244094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:48341 (local address 127.14.41.66:44873)
0925 10:08:06.244142 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:06.244144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:06.244266 (+ 122us) server_negotiation.cc:407] Connection header received
0925 10:08:06.244327 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:06.244328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:06.244339 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:06.244408 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:06.244583 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.245074 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.246116 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.246506 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.246594 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.246602 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:06.246783 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 10:08:06.246825 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:08:06.246905 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.244113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:34993 (local address 127.14.41.65:36441)
0925 10:08:06.244226 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:06.244239 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:06.244309 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:06.244484 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:06.244484 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:06.244488 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:06.244570 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.244572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.245214 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.245215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.246018 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.246021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.246572 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.246573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.246647 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.246651 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:08:06.246704 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 10:08:06.246735 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":1,"thread_start_us":56,"threads_started":1}
I20260925 10:08:06.246915 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.244031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:44873 (local address 127.14.41.65:48341)
0925 10:08:06.244176 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:06.244187 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:06.244265 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:06.244426 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:06.244427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:06.244430 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:06.244511 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.244514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.245162 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.245163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.246020 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.246023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.246616 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.246617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.246704 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.246708 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:08:06.246766 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 10:08:06.246804 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":3,"thread_start_us":82,"threads_started":1}
I20260925 10:08:06.247280 16681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847fcc742a9049a094ac6961f7d7037e" candidate_uuid: "63f3197999764c8cadbdd94ede65a75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" is_pre_election: true
I20260925 10:08:06.247480 16681 raft_consensus.cc:2454] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63f3197999764c8cadbdd94ede65a75f in term 0.
I20260925 10:08:06.247527 16540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847fcc742a9049a094ac6961f7d7037e" candidate_uuid: "63f3197999764c8cadbdd94ede65a75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d114b21c51064f0292a65a3aa3c55b34" is_pre_election: true
I20260925 10:08:06.247707 16540 raft_consensus.cc:2454] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63f3197999764c8cadbdd94ede65a75f in term 0.
I20260925 10:08:06.247737 16332 leader_election.cc:302] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63f3197999764c8cadbdd94ede65a75f, af3a8c6420224a1e98f2e272f0a0aa4c; no voters:
I20260925 10:08:06.247867 16758 raft_consensus.cc:2781] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:06.247949 16758 raft_consensus.cc:482] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:06.247985 16758 raft_consensus.cc:3037] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:06.248732 16758 raft_consensus.cc:504] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.248891 16758 leader_election.cc:288] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [CANDIDATE]: Term 1 election: Requested vote from peers d114b21c51064f0292a65a3aa3c55b34 (127.14.41.66:44873), af3a8c6420224a1e98f2e272f0a0aa4c (127.14.41.67:34993)
I20260925 10:08:06.249058 16540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847fcc742a9049a094ac6961f7d7037e" candidate_uuid: "63f3197999764c8cadbdd94ede65a75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d114b21c51064f0292a65a3aa3c55b34"
I20260925 10:08:06.249127 16681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847fcc742a9049a094ac6961f7d7037e" candidate_uuid: "63f3197999764c8cadbdd94ede65a75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c"
I20260925 10:08:06.249141 16540 raft_consensus.cc:3037] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:06.249214 16681 raft_consensus.cc:3037] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:06.249941 16540 raft_consensus.cc:2454] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63f3197999764c8cadbdd94ede65a75f in term 1.
I20260925 10:08:06.250043 16681 raft_consensus.cc:2454] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63f3197999764c8cadbdd94ede65a75f in term 1.
I20260925 10:08:06.250138 16330 leader_election.cc:302] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63f3197999764c8cadbdd94ede65a75f, d114b21c51064f0292a65a3aa3c55b34; no voters:
I20260925 10:08:06.250233 16758 raft_consensus.cc:2781] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:06.250455 16758 raft_consensus.cc:686] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [term 1 LEADER]: Becoming Leader. State: Replica: 63f3197999764c8cadbdd94ede65a75f, State: Running, Role: LEADER
I20260925 10:08:06.250576 16758 consensus_queue.cc:237] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.251428 16242 catalog_manager.cc:5495] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f reported cstate change: term changed from 0 to 1, leader changed from <none> to 63f3197999764c8cadbdd94ede65a75f (127.14.41.65). New cstate: current_term: 1 leader_uuid: "63f3197999764c8cadbdd94ede65a75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } health_report { overall_health: UNKNOWN } } }
Sep 25 10:08:06 dist-test-slave-s6tm krb5kdc[16196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330885, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:06.280094 16768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.272644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:37185 (local address 127.0.0.1:59984)
0925 10:08:06.272866 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:06.272940 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:06.273032 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:06.273183 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:06.273184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:06.273546 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:06.273601 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.273603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.274345 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.274346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.274868 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.274870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.275219 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.275220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.275274 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.275815 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:06.275820 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:06.277257 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:06.278713 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:06.278720 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:06.278723 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:06.278967 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:06.279287 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:06.279287 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:06.279288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:06.279388 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:06.279664 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:06.279672 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:06.279722 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 10:08:06.279823 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 10:08:06.279970 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":135,"threads_started":1}
I20260925 10:08:06.280169 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.272719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59984 (local address 127.14.41.126:37185)
0925 10:08:06.272816 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:06.272818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:06.273010 (+ 192us) server_negotiation.cc:407] Connection header received
0925 10:08:06.273106 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:06.273106 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:06.273117 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:06.273175 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:06.273686 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.274261 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.274961 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.275146 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.275222 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.277350 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:06.277352 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:06.277352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:06.277359 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:06.278615 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:06.279118 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:06.279119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:06.279120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:06.279206 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:06.279463 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:06.279464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:06.279464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:06.279591 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:06.279676 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:06.279991 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 10:08:06.280060 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 10:08:06.282217 16242 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:59984:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 10:08:06.282406 16242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:08:06.284277 16520 tablet_service.cc:1467] Processing CreateTablet for tablet f47671e89c124df596e216dc1d448d5d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d7cff6c944f47d0b71da085690e7e30]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:08:06.284344 16380 tablet_service.cc:1467] Processing CreateTablet for tablet f47671e89c124df596e216dc1d448d5d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d7cff6c944f47d0b71da085690e7e30]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:08:06.284461 16520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f47671e89c124df596e216dc1d448d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:06.284538 16380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f47671e89c124df596e216dc1d448d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:06.285703 16750 tablet_bootstrap.cc:492] T f47671e89c124df596e216dc1d448d5d P d114b21c51064f0292a65a3aa3c55b34: Bootstrap starting.
I20260925 10:08:06.285806 16754 tablet_bootstrap.cc:492] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f: Bootstrap starting.
I20260925 10:08:06.286327 16750 tablet_bootstrap.cc:654] T f47671e89c124df596e216dc1d448d5d P d114b21c51064f0292a65a3aa3c55b34: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:06.286415 16754 tablet_bootstrap.cc:654] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:06.286826 16661 tablet_service.cc:1467] Processing CreateTablet for tablet f47671e89c124df596e216dc1d448d5d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d7cff6c944f47d0b71da085690e7e30]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:08:06.287005 16750 tablet_bootstrap.cc:492] T f47671e89c124df596e216dc1d448d5d P d114b21c51064f0292a65a3aa3c55b34: No bootstrap required, opened a new log
I20260925 10:08:06.287010 16661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f47671e89c124df596e216dc1d448d5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:06.287057 16750 ts_tablet_manager.cc:1397] T f47671e89c124df596e216dc1d448d5d P d114b21c51064f0292a65a3aa3c55b34: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:06.287081 16754 tablet_bootstrap.cc:492] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f: No bootstrap required, opened a new log
I20260925 10:08:06.287144 16754 ts_tablet_manager.cc:1397] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:06.287220 16750 raft_consensus.cc:348] T f47671e89c124df596e216dc1d448d5d P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.287320 16750 raft_consensus.cc:374] T f47671e89c124df596e216dc1d448d5d P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:06.287288 16754 raft_consensus.cc:348] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.287352 16754 raft_consensus.cc:374] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:06.287351 16750 raft_consensus.cc:729] T f47671e89c124df596e216dc1d448d5d P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d114b21c51064f0292a65a3aa3c55b34, State: Initialized, Role: FOLLOWER
I20260925 10:08:06.287371 16754 raft_consensus.cc:729] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63f3197999764c8cadbdd94ede65a75f, State: Initialized, Role: FOLLOWER
I20260925 10:08:06.287389 16750 consensus_queue.cc:260] T f47671e89c124df596e216dc1d448d5d P d114b21c51064f0292a65a3aa3c55b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.287420 16754 consensus_queue.cc:260] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.287484 16750 ts_tablet_manager.cc:1428] T f47671e89c124df596e216dc1d448d5d P d114b21c51064f0292a65a3aa3c55b34: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:06.287503 16754 ts_tablet_manager.cc:1428] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:08:06.287550 16753 tablet_replica.cc:439] T f47671e89c124df596e216dc1d448d5d P d114b21c51064f0292a65a3aa3c55b34: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } } }
I20260925 10:08:06.287627 16753 tablet_replica.cc:442] T f47671e89c124df596e216dc1d448d5d P d114b21c51064f0292a65a3aa3c55b34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:08:06.288669 16751 tablet_bootstrap.cc:492] T f47671e89c124df596e216dc1d448d5d P af3a8c6420224a1e98f2e272f0a0aa4c: Bootstrap starting.
I20260925 10:08:06.288719 16758 tablet_replica.cc:439] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } } }
I20260925 10:08:06.288829 16758 tablet_replica.cc:442] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:08:06.289357 16751 tablet_bootstrap.cc:654] T f47671e89c124df596e216dc1d448d5d P af3a8c6420224a1e98f2e272f0a0aa4c: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:06.290190 16751 tablet_bootstrap.cc:492] T f47671e89c124df596e216dc1d448d5d P af3a8c6420224a1e98f2e272f0a0aa4c: No bootstrap required, opened a new log
I20260925 10:08:06.290237 16751 ts_tablet_manager.cc:1397] T f47671e89c124df596e216dc1d448d5d P af3a8c6420224a1e98f2e272f0a0aa4c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:08:06.290405 16751 raft_consensus.cc:348] T f47671e89c124df596e216dc1d448d5d P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.290488 16751 raft_consensus.cc:374] T f47671e89c124df596e216dc1d448d5d P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:06.290526 16751 raft_consensus.cc:729] T f47671e89c124df596e216dc1d448d5d P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3a8c6420224a1e98f2e272f0a0aa4c, State: Initialized, Role: FOLLOWER
I20260925 10:08:06.290584 16751 consensus_queue.cc:260] T f47671e89c124df596e216dc1d448d5d P af3a8c6420224a1e98f2e272f0a0aa4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.290700 16751 ts_tablet_manager.cc:1428] T f47671e89c124df596e216dc1d448d5d P af3a8c6420224a1e98f2e272f0a0aa4c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:08:06.290805 16757 tablet_replica.cc:439] T f47671e89c124df596e216dc1d448d5d P af3a8c6420224a1e98f2e272f0a0aa4c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } } }
I20260925 10:08:06.290906 16757 tablet_replica.cc:442] T f47671e89c124df596e216dc1d448d5d P af3a8c6420224a1e98f2e272f0a0aa4c: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 10:08:06.315555 16728 tablet.cc:2369] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:08:06.399713 16758 raft_consensus.cc:482] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:06.399948 16758 raft_consensus.cc:504] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.400153 16758 leader_election.cc:288] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d114b21c51064f0292a65a3aa3c55b34 (127.14.41.66:44873), af3a8c6420224a1e98f2e272f0a0aa4c (127.14.41.67:34993)
I20260925 10:08:06.400472 16681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47671e89c124df596e216dc1d448d5d" candidate_uuid: "63f3197999764c8cadbdd94ede65a75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" is_pre_election: true
I20260925 10:08:06.400439 16540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47671e89c124df596e216dc1d448d5d" candidate_uuid: "63f3197999764c8cadbdd94ede65a75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d114b21c51064f0292a65a3aa3c55b34" is_pre_election: true
I20260925 10:08:06.400574 16540 raft_consensus.cc:2454] T f47671e89c124df596e216dc1d448d5d P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63f3197999764c8cadbdd94ede65a75f in term 0.
I20260925 10:08:06.400599 16681 raft_consensus.cc:2454] T f47671e89c124df596e216dc1d448d5d P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63f3197999764c8cadbdd94ede65a75f in term 0.
I20260925 10:08:06.400733 16330 leader_election.cc:302] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63f3197999764c8cadbdd94ede65a75f, d114b21c51064f0292a65a3aa3c55b34; no voters:
I20260925 10:08:06.400864 16758 raft_consensus.cc:2781] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:06.400965 16758 raft_consensus.cc:482] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:06.400985 16758 raft_consensus.cc:3037] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:06.401721 16758 raft_consensus.cc:504] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.401894 16758 leader_election.cc:288] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f [CANDIDATE]: Term 1 election: Requested vote from peers d114b21c51064f0292a65a3aa3c55b34 (127.14.41.66:44873), af3a8c6420224a1e98f2e272f0a0aa4c (127.14.41.67:34993)
I20260925 10:08:06.402057 16540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47671e89c124df596e216dc1d448d5d" candidate_uuid: "63f3197999764c8cadbdd94ede65a75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d114b21c51064f0292a65a3aa3c55b34"
I20260925 10:08:06.402136 16540 raft_consensus.cc:3037] T f47671e89c124df596e216dc1d448d5d P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:06.402153 16681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47671e89c124df596e216dc1d448d5d" candidate_uuid: "63f3197999764c8cadbdd94ede65a75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c"
I20260925 10:08:06.402252 16681 raft_consensus.cc:3037] T f47671e89c124df596e216dc1d448d5d P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:06.402805 16540 raft_consensus.cc:2454] T f47671e89c124df596e216dc1d448d5d P d114b21c51064f0292a65a3aa3c55b34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63f3197999764c8cadbdd94ede65a75f in term 1.
I20260925 10:08:06.402959 16681 raft_consensus.cc:2454] T f47671e89c124df596e216dc1d448d5d P af3a8c6420224a1e98f2e272f0a0aa4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63f3197999764c8cadbdd94ede65a75f in term 1.
I20260925 10:08:06.402981 16330 leader_election.cc:302] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63f3197999764c8cadbdd94ede65a75f, d114b21c51064f0292a65a3aa3c55b34; no voters:
I20260925 10:08:06.403087 16758 raft_consensus.cc:2781] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:06.403177 16758 raft_consensus.cc:686] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f [term 1 LEADER]: Becoming Leader. State: Replica: 63f3197999764c8cadbdd94ede65a75f, State: Running, Role: LEADER
I20260925 10:08:06.403226 16758 consensus_queue.cc:237] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.403522 16761 tablet_replica.cc:439] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f: TxnStatusTablet state changed. Reason: New leader 63f3197999764c8cadbdd94ede65a75f. Latest consensus state: current_term: 1 leader_uuid: "63f3197999764c8cadbdd94ede65a75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } } }
I20260925 10:08:06.403610 16761 tablet_replica.cc:442] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:08:06.403824 16773 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 10:08:06.404006 16773 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 10:08:06.403995 16241 catalog_manager.cc:5495] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f reported cstate change: term changed from 0 to 1, leader changed from <none> to 63f3197999764c8cadbdd94ede65a75f (127.14.41.65). New cstate: current_term: 1 leader_uuid: "63f3197999764c8cadbdd94ede65a75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } health_report { overall_health: UNKNOWN } } }
I20260925 10:08:06.436617 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.433728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55632 (local address 127.14.41.65:34163)
0925 10:08:06.433799 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:06.433801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:06.433811 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:08:06.433853 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:06.433854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:06.433877 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:06.433973 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:06.434131 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.434892 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.435649 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.435980 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.436048 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.436222 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:06.436371 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:06.436452 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:06.436510 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 10:08:06.436532 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 10:08:06.436612 16768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.433612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:34163 (local address 127.0.0.1:55632)
0925 10:08:06.433700 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:06.433715 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:06.433779 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:06.433993 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:06.434005 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:06.434007 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:08:06.434059 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.434062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.434977 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.434978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.435565 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.435566 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.436055 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.436056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.436111 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.436126 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:08:06.436450 (+ 324us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:08:06.436460 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 10:08:06.436497 (+ 37us) client_negotiation.cc:240] Negotiation successful
0925 10:08:06.436551 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 10:08:06.438791 16681 raft_consensus.cc:1261] T f47671e89c124df596e216dc1d448d5d P af3a8c6420224a1e98f2e272f0a0aa4c [term 1 FOLLOWER]: Refusing update from remote peer 63f3197999764c8cadbdd94ede65a75f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:06.438786 16540 raft_consensus.cc:1261] T f47671e89c124df596e216dc1d448d5d P d114b21c51064f0292a65a3aa3c55b34 [term 1 FOLLOWER]: Refusing update from remote peer 63f3197999764c8cadbdd94ede65a75f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:06.439149 16758 consensus_queue.cc:1035] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:06.439285 16761 consensus_queue.cc:1035] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:06.440524 16757 tablet_replica.cc:439] T f47671e89c124df596e216dc1d448d5d P af3a8c6420224a1e98f2e272f0a0aa4c: TxnStatusTablet state changed. Reason: New leader 63f3197999764c8cadbdd94ede65a75f. Latest consensus state: current_term: 1 leader_uuid: "63f3197999764c8cadbdd94ede65a75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } } }
I20260925 10:08:06.440642 16757 tablet_replica.cc:442] T f47671e89c124df596e216dc1d448d5d P af3a8c6420224a1e98f2e272f0a0aa4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:08:06.441624 16753 tablet_replica.cc:439] T f47671e89c124df596e216dc1d448d5d P d114b21c51064f0292a65a3aa3c55b34: TxnStatusTablet state changed. Reason: New leader 63f3197999764c8cadbdd94ede65a75f. Latest consensus state: current_term: 1 leader_uuid: "63f3197999764c8cadbdd94ede65a75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } } }
I20260925 10:08:06.441722 16753 tablet_replica.cc:442] T f47671e89c124df596e216dc1d448d5d P d114b21c51064f0292a65a3aa3c55b34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:08:06.441813 16761 tablet_replica.cc:439] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63f3197999764c8cadbdd94ede65a75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } } }
I20260925 10:08:06.441898 16761 tablet_replica.cc:442] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:08:06.442137 16776 mvcc.cc:205] Tried to move back new op lower bound from 7333195310849978368 to 7333195310708006912. Current Snapshot: MvccSnapshot[applied={T|T < 7333195310849978368}]
I20260925 10:08:06.442433 16758 tablet_replica.cc:439] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "63f3197999764c8cadbdd94ede65a75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } } }
I20260925 10:08:06.442504 16758 tablet_replica.cc:442] T f47671e89c124df596e216dc1d448d5d P 63f3197999764c8cadbdd94ede65a75f: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:08:06.443351 16753 tablet_replica.cc:439] T f47671e89c124df596e216dc1d448d5d P d114b21c51064f0292a65a3aa3c55b34: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63f3197999764c8cadbdd94ede65a75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } } }
I20260925 10:08:06.443504 16753 tablet_replica.cc:442] T f47671e89c124df596e216dc1d448d5d P d114b21c51064f0292a65a3aa3c55b34: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:08:06.443578 16757 tablet_replica.cc:439] T f47671e89c124df596e216dc1d448d5d P af3a8c6420224a1e98f2e272f0a0aa4c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "63f3197999764c8cadbdd94ede65a75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } } }
I20260925 10:08:06.443653 16757 tablet_replica.cc:442] T f47671e89c124df596e216dc1d448d5d P af3a8c6420224a1e98f2e272f0a0aa4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:08:06.443761 16777 mvcc.cc:205] Tried to move back new op lower bound from 7333195310849978368 to 7333195310708006912. Current Snapshot: MvccSnapshot[applied={T|T < 7333195310849978368}]
I20260925 10:08:06.444670 16779 mvcc.cc:205] Tried to move back new op lower bound from 7333195310849978368 to 7333195310708006912. Current Snapshot: MvccSnapshot[applied={T|T < 7333195310849978368}]
I20260925 10:08:06.447654 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.444625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55646 (local address 127.14.41.65:34163)
0925 10:08:06.444817 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:06.444819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:06.444826 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:08:06.444847 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:06.444848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:06.444857 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:06.444899 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:06.445216 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.445972 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.446762 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.447065 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.447138 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.447334 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:06.447371 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:06.447433 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:06.447551 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 10:08:06.447573 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260925 10:08:06.453543 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.450503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:39051 (local address 127.14.41.65:34163)
0925 10:08:06.450554 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:06.450556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:06.450673 (+ 117us) server_negotiation.cc:407] Connection header received
0925 10:08:06.450702 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:06.450703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:06.450712 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:06.450794 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:06.451013 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.451725 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.452716 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.453027 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.453099 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.453290 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:06.453323 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:06.453375 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:06.453429 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 10:08:06.453447 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 10:08:06.453548 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.450406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:34163 (local address 127.14.41.65:39051)
0925 10:08:06.450593 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:06.450614 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:06.450670 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:06.450804 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:06.450805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:06.450807 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:08:06.450947 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.450949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.451997 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.452000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.452634 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.452637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.453123 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.453124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.453181 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.453196 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:08:06.453389 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:08:06.453391 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:08:06.453427 (+ 36us) client_negotiation.cc:240] Negotiation successful
0925 10:08:06.453466 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":121,"threads_started":1}
I20260925 10:08:06.456183 16540 raft_consensus.cc:1261] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34 [term 1 FOLLOWER]: Refusing update from remote peer 63f3197999764c8cadbdd94ede65a75f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:06.456274 16681 raft_consensus.cc:1261] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c [term 1 FOLLOWER]: Refusing update from remote peer 63f3197999764c8cadbdd94ede65a75f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:06.456391 16758 consensus_queue.cc:1035] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:06.456516 16788 consensus_queue.cc:1035] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:06.472512 16241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:06.472656 16241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:06.473587 16241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526: Sending DeleteTablet for 3 replicas of tablet 847fcc742a9049a094ac6961f7d7037e
I20260925 10:08:06.473961 16520 tablet_service.cc:1514] Processing DeleteTablet for tablet 847fcc742a9049a094ac6961f7d7037e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:06 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:50486
I20260925 10:08:06.474086 16661 tablet_service.cc:1514] Processing DeleteTablet for tablet 847fcc742a9049a094ac6961f7d7037e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:06 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:50456
I20260925 10:08:06.474229 16793 tablet_replica.cc:330] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34: stopping tablet replica
I20260925 10:08:06.474339 16793 raft_consensus.cc:2229] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:06.474414 16793 raft_consensus.cc:2258] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:06.474386 16241 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:08:06.474553 16241 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:08:06.474743 16380 tablet_service.cc:1514] Processing DeleteTablet for tablet 847fcc742a9049a094ac6961f7d7037e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:06 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:55622
I20260925 10:08:06.474911 16793 ts_tablet_manager.cc:1905] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:06.475992 16380 tablet_service.cc:1467] Processing CreateTablet for tablet 8016345a1f4c4bf6aa67b17987476c84 (DEFAULT_TABLE table=test-table [id=91d7e458dfcc4c1893341e9f86bccc77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:06.475998 16661 tablet_service.cc:1467] Processing CreateTablet for tablet 8016345a1f4c4bf6aa67b17987476c84 (DEFAULT_TABLE table=test-table [id=91d7e458dfcc4c1893341e9f86bccc77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:06.476117 16380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8016345a1f4c4bf6aa67b17987476c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:06.476119 16661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8016345a1f4c4bf6aa67b17987476c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:06.476812 16793 ts_tablet_manager.cc:1918] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:08:06.476883 16793 log.cc:1196] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/847fcc742a9049a094ac6961f7d7037e
I20260925 10:08:06.477136 16793 ts_tablet_manager.cc:1939] T 847fcc742a9049a094ac6961f7d7037e P d114b21c51064f0292a65a3aa3c55b34: Deleting consensus metadata
I20260925 10:08:06.477444 16754 tablet_bootstrap.cc:492] T 8016345a1f4c4bf6aa67b17987476c84 P 63f3197999764c8cadbdd94ede65a75f: Bootstrap starting.
I20260925 10:08:06.477444 16751 tablet_bootstrap.cc:492] T 8016345a1f4c4bf6aa67b17987476c84 P af3a8c6420224a1e98f2e272f0a0aa4c: Bootstrap starting.
I20260925 10:08:06.477769 16754 tablet_bootstrap.cc:654] T 8016345a1f4c4bf6aa67b17987476c84 P 63f3197999764c8cadbdd94ede65a75f: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:06.477856 16751 tablet_bootstrap.cc:654] T 8016345a1f4c4bf6aa67b17987476c84 P af3a8c6420224a1e98f2e272f0a0aa4c: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:06.478449 16226 catalog_manager.cc:4841] TS d114b21c51064f0292a65a3aa3c55b34 (127.14.41.66:44873): tablet 847fcc742a9049a094ac6961f7d7037e (table test-table [id=6ecfd6da7aac493185d8273d5fed836f]) successfully deleted
I20260925 10:08:06.478595 16520 tablet_service.cc:1467] Processing CreateTablet for tablet 8016345a1f4c4bf6aa67b17987476c84 (DEFAULT_TABLE table=test-table [id=91d7e458dfcc4c1893341e9f86bccc77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:06.478693 16754 tablet_bootstrap.cc:492] T 8016345a1f4c4bf6aa67b17987476c84 P 63f3197999764c8cadbdd94ede65a75f: No bootstrap required, opened a new log
I20260925 10:08:06.478708 16520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8016345a1f4c4bf6aa67b17987476c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:06.478757 16754 ts_tablet_manager.cc:1397] T 8016345a1f4c4bf6aa67b17987476c84 P 63f3197999764c8cadbdd94ede65a75f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:06.478907 16754 raft_consensus.cc:348] T 8016345a1f4c4bf6aa67b17987476c84 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.479015 16754 raft_consensus.cc:374] T 8016345a1f4c4bf6aa67b17987476c84 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:06.479043 16754 raft_consensus.cc:729] T 8016345a1f4c4bf6aa67b17987476c84 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63f3197999764c8cadbdd94ede65a75f, State: Initialized, Role: FOLLOWER
I20260925 10:08:06.479086 16754 consensus_queue.cc:260] T 8016345a1f4c4bf6aa67b17987476c84 P 63f3197999764c8cadbdd94ede65a75f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.479185 16754 ts_tablet_manager.cc:1428] T 8016345a1f4c4bf6aa67b17987476c84 P 63f3197999764c8cadbdd94ede65a75f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:06.475117 16795 tablet_replica.cc:330] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f: stopping tablet replica
I20260925 10:08:06.480042 16794 tablet_replica.cc:330] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c: stopping tablet replica
I20260925 10:08:06.480103 16750 tablet_bootstrap.cc:492] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34: Bootstrap starting.
I20260925 10:08:06.480249 16794 raft_consensus.cc:2229] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:06.480374 16794 raft_consensus.cc:2258] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:06.480033 16795 raft_consensus.cc:2229] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:08:06.480513 16750 tablet_bootstrap.cc:654] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:06.480582 16795 raft_consensus.cc:2258] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:06.480985 16751 tablet_bootstrap.cc:492] T 8016345a1f4c4bf6aa67b17987476c84 P af3a8c6420224a1e98f2e272f0a0aa4c: No bootstrap required, opened a new log
I20260925 10:08:06.481040 16751 ts_tablet_manager.cc:1397] T 8016345a1f4c4bf6aa67b17987476c84 P af3a8c6420224a1e98f2e272f0a0aa4c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 10:08:06.481046 16795 ts_tablet_manager.cc:1905] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:06.481177 16794 ts_tablet_manager.cc:1905] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:06.481202 16751 raft_consensus.cc:348] T 8016345a1f4c4bf6aa67b17987476c84 P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.481281 16751 raft_consensus.cc:374] T 8016345a1f4c4bf6aa67b17987476c84 P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:06.481307 16751 raft_consensus.cc:729] T 8016345a1f4c4bf6aa67b17987476c84 P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3a8c6420224a1e98f2e272f0a0aa4c, State: Initialized, Role: FOLLOWER
I20260925 10:08:06.481357 16751 consensus_queue.cc:260] T 8016345a1f4c4bf6aa67b17987476c84 P af3a8c6420224a1e98f2e272f0a0aa4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.481449 16751 ts_tablet_manager.cc:1428] T 8016345a1f4c4bf6aa67b17987476c84 P af3a8c6420224a1e98f2e272f0a0aa4c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:08:06.483327 16794 ts_tablet_manager.cc:1918] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:08:06.483374 16795 ts_tablet_manager.cc:1918] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:08:06.483417 16794 log.cc:1196] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/847fcc742a9049a094ac6961f7d7037e
I20260925 10:08:06.483433 16795 log.cc:1196] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-0/wal/wals/847fcc742a9049a094ac6961f7d7037e
I20260925 10:08:06.483690 16795 ts_tablet_manager.cc:1939] T 847fcc742a9049a094ac6961f7d7037e P 63f3197999764c8cadbdd94ede65a75f: Deleting consensus metadata
I20260925 10:08:06.483690 16794 ts_tablet_manager.cc:1939] T 847fcc742a9049a094ac6961f7d7037e P af3a8c6420224a1e98f2e272f0a0aa4c: Deleting consensus metadata
I20260925 10:08:06.484153 16228 catalog_manager.cc:4841] TS 63f3197999764c8cadbdd94ede65a75f (127.14.41.65:34163): tablet 847fcc742a9049a094ac6961f7d7037e (table test-table [id=6ecfd6da7aac493185d8273d5fed836f]) successfully deleted
I20260925 10:08:06.484263 16228 catalog_manager.cc:4841] TS af3a8c6420224a1e98f2e272f0a0aa4c (127.14.41.67:34993): tablet 847fcc742a9049a094ac6961f7d7037e (table test-table [id=6ecfd6da7aac493185d8273d5fed836f]) successfully deleted
I20260925 10:08:06.487397 16750 tablet_bootstrap.cc:492] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34: No bootstrap required, opened a new log
I20260925 10:08:06.487452 16750 ts_tablet_manager.cc:1397] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260925 10:08:06.487629 16750 raft_consensus.cc:348] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.487692 16750 raft_consensus.cc:374] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:06.487706 16750 raft_consensus.cc:729] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d114b21c51064f0292a65a3aa3c55b34, State: Initialized, Role: FOLLOWER
I20260925 10:08:06.487776 16750 consensus_queue.cc:260] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.487915 16750 ts_tablet_manager.cc:1428] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:06.510071 16753 raft_consensus.cc:482] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:06.510169 16753 raft_consensus.cc:504] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.510574 16753 leader_election.cc:288] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63f3197999764c8cadbdd94ede65a75f (127.14.41.65:34163), af3a8c6420224a1e98f2e272f0a0aa4c (127.14.41.67:34993)
I20260925 10:08:06.514428 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.511028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:39093 (local address 127.14.41.67:34993)
0925 10:08:06.511073 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:06.511075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:06.511187 (+ 112us) server_negotiation.cc:407] Connection header received
0925 10:08:06.511300 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:06.511301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:06.511311 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:06.511340 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:06.511571 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.512289 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.513428 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.514032 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.514107 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.514117 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:06.514291 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 10:08:06.514339 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:08:06.514585 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.510838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:59825 (local address 127.14.41.65:34163)
0925 10:08:06.510913 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:06.510915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:06.511032 (+ 117us) server_negotiation.cc:407] Connection header received
0925 10:08:06.511184 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:06.511185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:06.511195 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:06.511248 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:06.511412 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.512183 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.513625 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.514172 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.514263 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.514273 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:06.514448 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 10:08:06.514484 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:08:06.514667 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.510933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:34993 (local address 127.14.41.66:39093)
0925 10:08:06.511122 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:06.511132 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:06.511219 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:06.511417 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:06.511418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:06.511419 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:06.511477 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.511478 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.512374 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.512376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.513333 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.513336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.514122 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.514123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.514187 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.514192 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:08:06.514530 (+ 338us) client_negotiation.cc:240] Negotiation successful
0925 10:08:06.514574 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":107,"threads_started":1}
I20260925 10:08:06.514938 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.510764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:34163 (local address 127.14.41.66:59825)
0925 10:08:06.510959 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:06.510971 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:06.511064 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:06.511251 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:06.511252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:06.511254 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:06.511342 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.511344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.512277 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.512279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.513525 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.513528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.514297 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.514298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.514372 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.514375 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:06.514442 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 10:08:06.514482 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":64,"threads_started":1}
I20260925 10:08:06.515178 16681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8016345a1f4c4bf6aa67b17987476c84" candidate_uuid: "d114b21c51064f0292a65a3aa3c55b34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" is_pre_election: true
I20260925 10:08:06.515282 16681 raft_consensus.cc:2454] T 8016345a1f4c4bf6aa67b17987476c84 P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d114b21c51064f0292a65a3aa3c55b34 in term 0.
I20260925 10:08:06.515267 16400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8016345a1f4c4bf6aa67b17987476c84" candidate_uuid: "d114b21c51064f0292a65a3aa3c55b34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63f3197999764c8cadbdd94ede65a75f" is_pre_election: true
I20260925 10:08:06.515417 16400 raft_consensus.cc:2454] T 8016345a1f4c4bf6aa67b17987476c84 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d114b21c51064f0292a65a3aa3c55b34 in term 0.
I20260925 10:08:06.515582 16473 leader_election.cc:302] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af3a8c6420224a1e98f2e272f0a0aa4c, d114b21c51064f0292a65a3aa3c55b34; no voters:
I20260925 10:08:06.515787 16753 raft_consensus.cc:2781] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:06.515841 16753 raft_consensus.cc:482] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:06.515868 16753 raft_consensus.cc:3037] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:06.516583 16753 raft_consensus.cc:504] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.516729 16753 leader_election.cc:288] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [CANDIDATE]: Term 1 election: Requested vote from peers 63f3197999764c8cadbdd94ede65a75f (127.14.41.65:34163), af3a8c6420224a1e98f2e272f0a0aa4c (127.14.41.67:34993)
I20260925 10:08:06.516901 16681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8016345a1f4c4bf6aa67b17987476c84" candidate_uuid: "d114b21c51064f0292a65a3aa3c55b34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c"
I20260925 10:08:06.516920 16400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8016345a1f4c4bf6aa67b17987476c84" candidate_uuid: "d114b21c51064f0292a65a3aa3c55b34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63f3197999764c8cadbdd94ede65a75f"
I20260925 10:08:06.516976 16681 raft_consensus.cc:3037] T 8016345a1f4c4bf6aa67b17987476c84 P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:06.516983 16400 raft_consensus.cc:3037] T 8016345a1f4c4bf6aa67b17987476c84 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:06.517695 16400 raft_consensus.cc:2454] T 8016345a1f4c4bf6aa67b17987476c84 P 63f3197999764c8cadbdd94ede65a75f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d114b21c51064f0292a65a3aa3c55b34 in term 1.
I20260925 10:08:06.517695 16681 raft_consensus.cc:2454] T 8016345a1f4c4bf6aa67b17987476c84 P af3a8c6420224a1e98f2e272f0a0aa4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d114b21c51064f0292a65a3aa3c55b34 in term 1.
I20260925 10:08:06.517870 16473 leader_election.cc:302] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af3a8c6420224a1e98f2e272f0a0aa4c, d114b21c51064f0292a65a3aa3c55b34; no voters:
I20260925 10:08:06.517972 16753 raft_consensus.cc:2781] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:06.518127 16753 raft_consensus.cc:686] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [term 1 LEADER]: Becoming Leader. State: Replica: d114b21c51064f0292a65a3aa3c55b34, State: Running, Role: LEADER
I20260925 10:08:06.518204 16753 consensus_queue.cc:237] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:06.518960 16242 catalog_manager.cc:5495] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 reported cstate change: term changed from 0 to 1, leader changed from <none> to d114b21c51064f0292a65a3aa3c55b34 (127.14.41.66). New cstate: current_term: 1 leader_uuid: "d114b21c51064f0292a65a3aa3c55b34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } health_report { overall_health: UNKNOWN } } }
I20260925 10:08:06.531280 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.528627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50488 (local address 127.14.41.66:44873)
0925 10:08:06.528669 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:06.528671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:06.528678 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:08:06.528711 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:06.528712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:06.528732 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:06.528799 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:06.528958 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.529422 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.530413 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.530613 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.530668 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.530916 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:06.531014 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:06.531129 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:06.531177 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 10:08:06.531199 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:08:06.536983 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.534040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:37649 (local address 127.14.41.65:34163)
0925 10:08:06.534089 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:06.534091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:06.534239 (+ 148us) server_negotiation.cc:407] Connection header received
0925 10:08:06.534310 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:06.534311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:06.534320 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:06.534372 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:06.534544 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.535045 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.536176 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.536393 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.536460 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.536673 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:06.536706 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:06.536748 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:06.536831 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 10:08:06.536852 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:08:06.536996 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.533979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:34163 (local address 127.14.41.66:37649)
0925 10:08:06.534177 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:06.534190 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:06.534256 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:06.534394 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:06.534395 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:06.534397 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:08:06.534478 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.534480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.535128 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.535129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.536097 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.536100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.536494 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.536496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.536563 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.536583 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:08:06.536771 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:08:06.536773 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:08:06.536828 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 10:08:06.536877 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":102,"threads_started":1}
I20260925 10:08:06.540006 16681 raft_consensus.cc:1261] T 8016345a1f4c4bf6aa67b17987476c84 P af3a8c6420224a1e98f2e272f0a0aa4c [term 1 FOLLOWER]: Refusing update from remote peer d114b21c51064f0292a65a3aa3c55b34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:06.540130 16400 raft_consensus.cc:1261] T 8016345a1f4c4bf6aa67b17987476c84 P 63f3197999764c8cadbdd94ede65a75f [term 1 FOLLOWER]: Refusing update from remote peer d114b21c51064f0292a65a3aa3c55b34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:06.540354 16801 consensus_queue.cc:1035] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:06.540553 16753 consensus_queue.cc:1035] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:06.550120 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.547218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:44873 (local address 127.14.41.65:41521)
0925 10:08:06.547259 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:06.547272 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:06.547315 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:06.547511 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:06.547512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:06.547513 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:08:06.547570 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.547571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.548586 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.548587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.549139 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:06.549141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.549665 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.549667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:06.549727 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.549736 (+ 9us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:08:06.549943 (+ 207us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:08:06.549945 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:08:06.549984 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 10:08:06.550035 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 10:08:06.550333 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:06.547335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:41521 (local address 127.14.41.66:44873)
0925 10:08:06.547381 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:06.547383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:06.547390 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:08:06.547411 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:06.547412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:06.547422 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:06.547509 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:06.547656 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.548416 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.549290 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:06.549590 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:06.549652 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:06.549820 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:06.549856 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:06.549904 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:06.550237 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 10:08:06.550260 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 10:08:06.550619 16779 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 10:08:08.077055 16241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:08.077195 16241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:08.078225 16241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526: Sending DeleteTablet for 3 replicas of tablet 8016345a1f4c4bf6aa67b17987476c84
I20260925 10:08:08.078550 16380 tablet_service.cc:1514] Processing DeleteTablet for tablet 8016345a1f4c4bf6aa67b17987476c84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:08 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:55622
I20260925 10:08:08.078572 16520 tablet_service.cc:1514] Processing DeleteTablet for tablet 8016345a1f4c4bf6aa67b17987476c84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:08 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:50486
I20260925 10:08:08.078756 16661 tablet_service.cc:1514] Processing DeleteTablet for tablet 8016345a1f4c4bf6aa67b17987476c84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:08 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:50456
I20260925 10:08:08.078950 16831 tablet_replica.cc:330] T 8016345a1f4c4bf6aa67b17987476c84 P 63f3197999764c8cadbdd94ede65a75f: stopping tablet replica
I20260925 10:08:08.079005 16831 raft_consensus.cc:2229] T 8016345a1f4c4bf6aa67b17987476c84 P 63f3197999764c8cadbdd94ede65a75f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:08.079043 16831 raft_consensus.cc:2258] T 8016345a1f4c4bf6aa67b17987476c84 P 63f3197999764c8cadbdd94ede65a75f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:08.079061 16832 tablet_replica.cc:330] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34: stopping tablet replica
I20260925 10:08:08.079114 16832 raft_consensus.cc:2229] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:08:08.079308 16833 tablet_replica.cc:330] T 8016345a1f4c4bf6aa67b17987476c84 P af3a8c6420224a1e98f2e272f0a0aa4c: stopping tablet replica
I20260925 10:08:08.079358 16833 raft_consensus.cc:2229] T 8016345a1f4c4bf6aa67b17987476c84 P af3a8c6420224a1e98f2e272f0a0aa4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:08.079361 16831 ts_tablet_manager.cc:1905] T 8016345a1f4c4bf6aa67b17987476c84 P 63f3197999764c8cadbdd94ede65a75f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:08.079397 16833 raft_consensus.cc:2258] T 8016345a1f4c4bf6aa67b17987476c84 P af3a8c6420224a1e98f2e272f0a0aa4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:08.079458 16832 raft_consensus.cc:2258] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:08.079653 16833 ts_tablet_manager.cc:1905] T 8016345a1f4c4bf6aa67b17987476c84 P af3a8c6420224a1e98f2e272f0a0aa4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:08.079679 16832 ts_tablet_manager.cc:1905] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:08.081625 16833 ts_tablet_manager.cc:1918] T 8016345a1f4c4bf6aa67b17987476c84 P af3a8c6420224a1e98f2e272f0a0aa4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:08:08.081625 16832 ts_tablet_manager.cc:1918] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:08:08.081699 16833 log.cc:1196] T 8016345a1f4c4bf6aa67b17987476c84 P af3a8c6420224a1e98f2e272f0a0aa4c: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/8016345a1f4c4bf6aa67b17987476c84
I20260925 10:08:08.081699 16832 log.cc:1196] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/8016345a1f4c4bf6aa67b17987476c84
I20260925 10:08:08.081980 16833 ts_tablet_manager.cc:1939] T 8016345a1f4c4bf6aa67b17987476c84 P af3a8c6420224a1e98f2e272f0a0aa4c: Deleting consensus metadata
I20260925 10:08:08.081991 16832 ts_tablet_manager.cc:1939] T 8016345a1f4c4bf6aa67b17987476c84 P d114b21c51064f0292a65a3aa3c55b34: Deleting consensus metadata
I20260925 10:08:08.082425 16226 catalog_manager.cc:4841] TS d114b21c51064f0292a65a3aa3c55b34 (127.14.41.66:44873): tablet 8016345a1f4c4bf6aa67b17987476c84 (table test-table [id=91d7e458dfcc4c1893341e9f86bccc77]) successfully deleted
I20260925 10:08:08.082419 16228 catalog_manager.cc:4841] TS af3a8c6420224a1e98f2e272f0a0aa4c (127.14.41.67:34993): tablet 8016345a1f4c4bf6aa67b17987476c84 (table test-table [id=91d7e458dfcc4c1893341e9f86bccc77]) successfully deleted
I20260925 10:08:08.082902 16831 ts_tablet_manager.cc:1918] T 8016345a1f4c4bf6aa67b17987476c84 P 63f3197999764c8cadbdd94ede65a75f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:08:08.082983 16831 log.cc:1196] T 8016345a1f4c4bf6aa67b17987476c84 P 63f3197999764c8cadbdd94ede65a75f: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-0/wal/wals/8016345a1f4c4bf6aa67b17987476c84
I20260925 10:08:08.083190 16831 ts_tablet_manager.cc:1939] T 8016345a1f4c4bf6aa67b17987476c84 P 63f3197999764c8cadbdd94ede65a75f: Deleting consensus metadata
I20260925 10:08:08.083810 16228 catalog_manager.cc:4841] TS 63f3197999764c8cadbdd94ede65a75f (127.14.41.65:34163): tablet 8016345a1f4c4bf6aa67b17987476c84 (table test-table [id=91d7e458dfcc4c1893341e9f86bccc77]) successfully deleted
Sep 25 10:08:08 dist-test-slave-s6tm krb5kdc[16196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330888, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:08:08 dist-test-slave-s6tm krb5kdc[16196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330888, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:08.106913 16841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:08.098109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.14.41.126:37185)
0925 10:08:08.098320 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:08.098322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:08.098334 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:08:08.098410 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:08.098411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:08.098423 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:08.098503 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:08.099188 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:08.099795 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:08.100694 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:08.100927 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:08.100997 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:08.103857 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:08.103859 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:08.103859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:08.103867 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:08.105633 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:08.106034 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:08.106035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:08.106035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:08.106108 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:08.106331 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:08.106332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:08.106332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:08.106461 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:08.106549 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:08.106700 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 10:08:08.106780 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":108,"threads_started":1}
I20260925 10:08:08.108808 16241 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:08:08.109016 16241 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:08:08.110783 16380 tablet_service.cc:1467] Processing CreateTablet for tablet 5b19eeba29e84cca8acb7c09533f2ed0 (DEFAULT_TABLE table=test-table [id=83e692a03c3442219f5d5a567a271ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:08.110816 16661 tablet_service.cc:1467] Processing CreateTablet for tablet 5b19eeba29e84cca8acb7c09533f2ed0 (DEFAULT_TABLE table=test-table [id=83e692a03c3442219f5d5a567a271ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:08.110816 16520 tablet_service.cc:1467] Processing CreateTablet for tablet 5b19eeba29e84cca8acb7c09533f2ed0 (DEFAULT_TABLE table=test-table [id=83e692a03c3442219f5d5a567a271ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:08.110967 16380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b19eeba29e84cca8acb7c09533f2ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:08.110965 16520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b19eeba29e84cca8acb7c09533f2ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:08.110975 16661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b19eeba29e84cca8acb7c09533f2ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:08.112452 16842 tablet_bootstrap.cc:492] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f: Bootstrap starting.
I20260925 10:08:08.112664 16844 tablet_bootstrap.cc:492] T 5b19eeba29e84cca8acb7c09533f2ed0 P d114b21c51064f0292a65a3aa3c55b34: Bootstrap starting.
I20260925 10:08:08.112869 16843 tablet_bootstrap.cc:492] T 5b19eeba29e84cca8acb7c09533f2ed0 P af3a8c6420224a1e98f2e272f0a0aa4c: Bootstrap starting.
I20260925 10:08:08.112990 16842 tablet_bootstrap.cc:654] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:08.113319 16843 tablet_bootstrap.cc:654] T 5b19eeba29e84cca8acb7c09533f2ed0 P af3a8c6420224a1e98f2e272f0a0aa4c: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:08.113688 16842 tablet_bootstrap.cc:492] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f: No bootstrap required, opened a new log
I20260925 10:08:08.113762 16842 ts_tablet_manager.cc:1397] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:08.113947 16842 raft_consensus.cc:348] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:08.114010 16844 tablet_bootstrap.cc:654] T 5b19eeba29e84cca8acb7c09533f2ed0 P d114b21c51064f0292a65a3aa3c55b34: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:08.114056 16842 raft_consensus.cc:374] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:08.114090 16842 raft_consensus.cc:729] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63f3197999764c8cadbdd94ede65a75f, State: Initialized, Role: FOLLOWER
I20260925 10:08:08.114141 16842 consensus_queue.cc:260] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:08.114291 16842 ts_tablet_manager.cc:1428] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:08:08.114585 16843 tablet_bootstrap.cc:492] T 5b19eeba29e84cca8acb7c09533f2ed0 P af3a8c6420224a1e98f2e272f0a0aa4c: No bootstrap required, opened a new log
I20260925 10:08:08.114642 16843 ts_tablet_manager.cc:1397] T 5b19eeba29e84cca8acb7c09533f2ed0 P af3a8c6420224a1e98f2e272f0a0aa4c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:08:08.114815 16843 raft_consensus.cc:348] T 5b19eeba29e84cca8acb7c09533f2ed0 P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:08.114902 16843 raft_consensus.cc:374] T 5b19eeba29e84cca8acb7c09533f2ed0 P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:08.114928 16843 raft_consensus.cc:729] T 5b19eeba29e84cca8acb7c09533f2ed0 P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3a8c6420224a1e98f2e272f0a0aa4c, State: Initialized, Role: FOLLOWER
I20260925 10:08:08.115140 16844 tablet_bootstrap.cc:492] T 5b19eeba29e84cca8acb7c09533f2ed0 P d114b21c51064f0292a65a3aa3c55b34: No bootstrap required, opened a new log
I20260925 10:08:08.115211 16844 ts_tablet_manager.cc:1397] T 5b19eeba29e84cca8acb7c09533f2ed0 P d114b21c51064f0292a65a3aa3c55b34: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 10:08:08.115337 16844 raft_consensus.cc:348] T 5b19eeba29e84cca8acb7c09533f2ed0 P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:08.115427 16844 raft_consensus.cc:374] T 5b19eeba29e84cca8acb7c09533f2ed0 P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:08.115461 16844 raft_consensus.cc:729] T 5b19eeba29e84cca8acb7c09533f2ed0 P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d114b21c51064f0292a65a3aa3c55b34, State: Initialized, Role: FOLLOWER
I20260925 10:08:08.115522 16844 consensus_queue.cc:260] T 5b19eeba29e84cca8acb7c09533f2ed0 P d114b21c51064f0292a65a3aa3c55b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:08.115629 16844 ts_tablet_manager.cc:1428] T 5b19eeba29e84cca8acb7c09533f2ed0 P d114b21c51064f0292a65a3aa3c55b34: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:08.115706 16843 consensus_queue.cc:260] T 5b19eeba29e84cca8acb7c09533f2ed0 P af3a8c6420224a1e98f2e272f0a0aa4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:08.115972 16843 ts_tablet_manager.cc:1428] T 5b19eeba29e84cca8acb7c09533f2ed0 P af3a8c6420224a1e98f2e272f0a0aa4c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:08.313990 16810 raft_consensus.cc:482] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:08.314111 16810 raft_consensus.cc:504] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:08.314478 16810 leader_election.cc:288] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d114b21c51064f0292a65a3aa3c55b34 (127.14.41.66:44873), af3a8c6420224a1e98f2e272f0a0aa4c (127.14.41.67:34993)
I20260925 10:08:08.314765 16681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b19eeba29e84cca8acb7c09533f2ed0" candidate_uuid: "63f3197999764c8cadbdd94ede65a75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" is_pre_election: true
I20260925 10:08:08.314783 16540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b19eeba29e84cca8acb7c09533f2ed0" candidate_uuid: "63f3197999764c8cadbdd94ede65a75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d114b21c51064f0292a65a3aa3c55b34" is_pre_election: true
I20260925 10:08:08.314893 16540 raft_consensus.cc:2454] T 5b19eeba29e84cca8acb7c09533f2ed0 P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63f3197999764c8cadbdd94ede65a75f in term 0.
I20260925 10:08:08.314896 16681 raft_consensus.cc:2454] T 5b19eeba29e84cca8acb7c09533f2ed0 P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63f3197999764c8cadbdd94ede65a75f in term 0.
I20260925 10:08:08.315106 16332 leader_election.cc:302] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63f3197999764c8cadbdd94ede65a75f, af3a8c6420224a1e98f2e272f0a0aa4c; no voters:
I20260925 10:08:08.315249 16810 raft_consensus.cc:2781] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:08.315307 16810 raft_consensus.cc:482] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:08.315335 16810 raft_consensus.cc:3037] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:08.316247 16810 raft_consensus.cc:504] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:08.316434 16810 leader_election.cc:288] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [CANDIDATE]: Term 1 election: Requested vote from peers d114b21c51064f0292a65a3aa3c55b34 (127.14.41.66:44873), af3a8c6420224a1e98f2e272f0a0aa4c (127.14.41.67:34993)
I20260925 10:08:08.316643 16540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b19eeba29e84cca8acb7c09533f2ed0" candidate_uuid: "63f3197999764c8cadbdd94ede65a75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d114b21c51064f0292a65a3aa3c55b34"
I20260925 10:08:08.316642 16681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b19eeba29e84cca8acb7c09533f2ed0" candidate_uuid: "63f3197999764c8cadbdd94ede65a75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c"
I20260925 10:08:08.316725 16540 raft_consensus.cc:3037] T 5b19eeba29e84cca8acb7c09533f2ed0 P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:08.316748 16681 raft_consensus.cc:3037] T 5b19eeba29e84cca8acb7c09533f2ed0 P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:08.317394 16681 raft_consensus.cc:2454] T 5b19eeba29e84cca8acb7c09533f2ed0 P af3a8c6420224a1e98f2e272f0a0aa4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63f3197999764c8cadbdd94ede65a75f in term 1.
I20260925 10:08:08.317478 16540 raft_consensus.cc:2454] T 5b19eeba29e84cca8acb7c09533f2ed0 P d114b21c51064f0292a65a3aa3c55b34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63f3197999764c8cadbdd94ede65a75f in term 1.
I20260925 10:08:08.317602 16332 leader_election.cc:302] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63f3197999764c8cadbdd94ede65a75f, af3a8c6420224a1e98f2e272f0a0aa4c; no voters:
I20260925 10:08:08.317699 16810 raft_consensus.cc:2781] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:08.317782 16810 raft_consensus.cc:686] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [term 1 LEADER]: Becoming Leader. State: Replica: 63f3197999764c8cadbdd94ede65a75f, State: Running, Role: LEADER
I20260925 10:08:08.317859 16810 consensus_queue.cc:237] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:08.318686 16241 catalog_manager.cc:5495] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f reported cstate change: term changed from 0 to 1, leader changed from <none> to 63f3197999764c8cadbdd94ede65a75f (127.14.41.65). New cstate: current_term: 1 leader_uuid: "63f3197999764c8cadbdd94ede65a75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } health_report { overall_health: UNKNOWN } } }
I20260925 10:08:08.348482 16851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:08.345627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.14.41.65:34163)
0925 10:08:08.345784 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:08.345786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:08.345794 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:08:08.345825 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:08.345825 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:08.345835 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:08.345892 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:08.346034 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:08.346585 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:08.347569 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:08.347813 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:08.347890 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:08.348125 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:08.348158 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:08.348205 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:08.348303 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 10:08:08.348328 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260925 10:08:08.351488 16681 raft_consensus.cc:1261] T 5b19eeba29e84cca8acb7c09533f2ed0 P af3a8c6420224a1e98f2e272f0a0aa4c [term 1 FOLLOWER]: Refusing update from remote peer 63f3197999764c8cadbdd94ede65a75f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:08.351496 16540 raft_consensus.cc:1261] T 5b19eeba29e84cca8acb7c09533f2ed0 P d114b21c51064f0292a65a3aa3c55b34 [term 1 FOLLOWER]: Refusing update from remote peer 63f3197999764c8cadbdd94ede65a75f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:08.351734 16813 consensus_queue.cc:1035] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:08.351871 16813 consensus_queue.cc:1035] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:08.363112 16241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:08.363253 16241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:08.364216 16241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526: Sending DeleteTablet for 3 replicas of tablet 5b19eeba29e84cca8acb7c09533f2ed0
I20260925 10:08:08.364558 16520 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b19eeba29e84cca8acb7c09533f2ed0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:08 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:50486
I20260925 10:08:08.364634 16661 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b19eeba29e84cca8acb7c09533f2ed0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:08 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:50456
I20260925 10:08:08.364665 16832 tablet_replica.cc:330] T 5b19eeba29e84cca8acb7c09533f2ed0 P d114b21c51064f0292a65a3aa3c55b34: stopping tablet replica
I20260925 10:08:08.364624 16380 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b19eeba29e84cca8acb7c09533f2ed0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:08 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:55622
I20260925 10:08:08.364712 16832 raft_consensus.cc:2229] T 5b19eeba29e84cca8acb7c09533f2ed0 P d114b21c51064f0292a65a3aa3c55b34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:08.364763 16832 raft_consensus.cc:2258] T 5b19eeba29e84cca8acb7c09533f2ed0 P d114b21c51064f0292a65a3aa3c55b34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:08.364774 16833 tablet_replica.cc:330] T 5b19eeba29e84cca8acb7c09533f2ed0 P af3a8c6420224a1e98f2e272f0a0aa4c: stopping tablet replica
I20260925 10:08:08.364848 16831 tablet_replica.cc:330] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f: stopping tablet replica
I20260925 10:08:08.364872 16833 raft_consensus.cc:2229] T 5b19eeba29e84cca8acb7c09533f2ed0 P af3a8c6420224a1e98f2e272f0a0aa4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:08.364920 16833 raft_consensus.cc:2258] T 5b19eeba29e84cca8acb7c09533f2ed0 P af3a8c6420224a1e98f2e272f0a0aa4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:08.364919 16831 raft_consensus.cc:2229] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:08:08.364991 16831 raft_consensus.cc:2258] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:08.365156 16241 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:08:08.365331 16241 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:08:08.365348 16833 ts_tablet_manager.cc:1905] T 5b19eeba29e84cca8acb7c09533f2ed0 P af3a8c6420224a1e98f2e272f0a0aa4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:08.365365 16831 ts_tablet_manager.cc:1905] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:08.365388 16832 ts_tablet_manager.cc:1905] T 5b19eeba29e84cca8acb7c09533f2ed0 P d114b21c51064f0292a65a3aa3c55b34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:08.366708 16661 tablet_service.cc:1467] Processing CreateTablet for tablet b0dd8f755b9b4fde8ccdff1a421ea8f4 (DEFAULT_TABLE table=test-table [id=ce302c4b849d4eafb00c1ed58fbeaa0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:08.366817 16661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0dd8f755b9b4fde8ccdff1a421ea8f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:08.367240 16832 ts_tablet_manager.cc:1918] T 5b19eeba29e84cca8acb7c09533f2ed0 P d114b21c51064f0292a65a3aa3c55b34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:08:08.367296 16832 log.cc:1196] T 5b19eeba29e84cca8acb7c09533f2ed0 P d114b21c51064f0292a65a3aa3c55b34: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/5b19eeba29e84cca8acb7c09533f2ed0
I20260925 10:08:08.367350 16833 ts_tablet_manager.cc:1918] T 5b19eeba29e84cca8acb7c09533f2ed0 P af3a8c6420224a1e98f2e272f0a0aa4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:08:08.367411 16833 log.cc:1196] T 5b19eeba29e84cca8acb7c09533f2ed0 P af3a8c6420224a1e98f2e272f0a0aa4c: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/5b19eeba29e84cca8acb7c09533f2ed0
I20260925 10:08:08.367516 16831 ts_tablet_manager.cc:1918] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:08:08.367583 16831 log.cc:1196] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-0/wal/wals/5b19eeba29e84cca8acb7c09533f2ed0
I20260925 10:08:08.367667 16833 ts_tablet_manager.cc:1939] T 5b19eeba29e84cca8acb7c09533f2ed0 P af3a8c6420224a1e98f2e272f0a0aa4c: Deleting consensus metadata
I20260925 10:08:08.367674 16832 ts_tablet_manager.cc:1939] T 5b19eeba29e84cca8acb7c09533f2ed0 P d114b21c51064f0292a65a3aa3c55b34: Deleting consensus metadata
I20260925 10:08:08.367844 16831 ts_tablet_manager.cc:1939] T 5b19eeba29e84cca8acb7c09533f2ed0 P 63f3197999764c8cadbdd94ede65a75f: Deleting consensus metadata
I20260925 10:08:08.368209 16843 tablet_bootstrap.cc:492] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c: Bootstrap starting.
I20260925 10:08:08.368315 16228 catalog_manager.cc:4841] TS 63f3197999764c8cadbdd94ede65a75f (127.14.41.65:34163): tablet 5b19eeba29e84cca8acb7c09533f2ed0 (table test-table [id=83e692a03c3442219f5d5a567a271ca5]) successfully deleted
I20260925 10:08:08.368395 16380 tablet_service.cc:1467] Processing CreateTablet for tablet b0dd8f755b9b4fde8ccdff1a421ea8f4 (DEFAULT_TABLE table=test-table [id=ce302c4b849d4eafb00c1ed58fbeaa0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:08.368513 16380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0dd8f755b9b4fde8ccdff1a421ea8f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:08.368503 16228 catalog_manager.cc:4841] TS af3a8c6420224a1e98f2e272f0a0aa4c (127.14.41.67:34993): tablet 5b19eeba29e84cca8acb7c09533f2ed0 (table test-table [id=83e692a03c3442219f5d5a567a271ca5]) successfully deleted
I20260925 10:08:08.368615 16843 tablet_bootstrap.cc:654] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:08.368610 16520 tablet_service.cc:1467] Processing CreateTablet for tablet b0dd8f755b9b4fde8ccdff1a421ea8f4 (DEFAULT_TABLE table=test-table [id=ce302c4b849d4eafb00c1ed58fbeaa0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:08.368722 16520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0dd8f755b9b4fde8ccdff1a421ea8f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:08.369694 16842 tablet_bootstrap.cc:492] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f: Bootstrap starting.
I20260925 10:08:08.368315 16226 catalog_manager.cc:4841] TS d114b21c51064f0292a65a3aa3c55b34 (127.14.41.66:44873): tablet 5b19eeba29e84cca8acb7c09533f2ed0 (table test-table [id=83e692a03c3442219f5d5a567a271ca5]) successfully deleted
I20260925 10:08:08.370024 16844 tablet_bootstrap.cc:492] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34: Bootstrap starting.
I20260925 10:08:08.370090 16842 tablet_bootstrap.cc:654] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:08.370376 16844 tablet_bootstrap.cc:654] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:08.370798 16843 tablet_bootstrap.cc:492] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c: No bootstrap required, opened a new log
I20260925 10:08:08.370839 16843 ts_tablet_manager.cc:1397] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 10:08:08.370893 16842 tablet_bootstrap.cc:492] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f: No bootstrap required, opened a new log
I20260925 10:08:08.370944 16842 ts_tablet_manager.cc:1397] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:08.371034 16842 raft_consensus.cc:348] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:08.371065 16844 tablet_bootstrap.cc:492] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34: No bootstrap required, opened a new log
I20260925 10:08:08.371086 16842 raft_consensus.cc:374] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:08.371100 16842 raft_consensus.cc:729] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63f3197999764c8cadbdd94ede65a75f, State: Initialized, Role: FOLLOWER
I20260925 10:08:08.371037 16843 raft_consensus.cc:348] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:08.371109 16844 ts_tablet_manager.cc:1397] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:08.371133 16843 raft_consensus.cc:374] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:08.371160 16843 raft_consensus.cc:729] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3a8c6420224a1e98f2e272f0a0aa4c, State: Initialized, Role: FOLLOWER
I20260925 10:08:08.371168 16842 consensus_queue.cc:260] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:08.371223 16843 consensus_queue.cc:260] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:08.371224 16844 raft_consensus.cc:348] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:08.371277 16842 ts_tablet_manager.cc:1428] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:08.371310 16844 raft_consensus.cc:374] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:08.371337 16844 raft_consensus.cc:729] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d114b21c51064f0292a65a3aa3c55b34, State: Initialized, Role: FOLLOWER
I20260925 10:08:08.371330 16843 ts_tablet_manager.cc:1428] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:08:08.371392 16844 consensus_queue.cc:260] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:08.371516 16844 ts_tablet_manager.cc:1428] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:08.475878 16848 raft_consensus.cc:482] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:08.476016 16848 raft_consensus.cc:504] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:08.476441 16848 leader_election.cc:288] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63f3197999764c8cadbdd94ede65a75f (127.14.41.65:34163), d114b21c51064f0292a65a3aa3c55b34 (127.14.41.66:44873)
I20260925 10:08:08.480191 16859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:08.476687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:44873 (local address 127.14.41.67:39609)
0925 10:08:08.476889 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:08.476898 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:08.476956 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:08.477150 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:08.477151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:08.477153 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:08.477231 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:08.477233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:08.477994 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:08.477995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:08.479006 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:08.479009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:08.479854 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:08.479855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:08.479969 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:08.479974 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:08:08.480029 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 10:08:08.480086 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":1,"thread_start_us":132,"threads_started":1}
I20260925 10:08:08.480212 16810 raft_consensus.cc:482] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:08.480224 16851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:08.476831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:49219 (local address 127.14.41.65:34163)
0925 10:08:08.476997 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:08.476999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:08.477008 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:08:08.477045 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:08.477046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:08.477058 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:08.477129 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:08.477433 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:08.477902 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:08.479130 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:08.479704 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:08.479791 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:08.479802 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:08.480097 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 10:08:08.480150 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260925 10:08:08.480304 16810 raft_consensus.cc:504] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:08.480502 16810 leader_election.cc:288] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d114b21c51064f0292a65a3aa3c55b34 (127.14.41.66:44873), af3a8c6420224a1e98f2e272f0a0aa4c (127.14.41.67:34993)
I20260925 10:08:08.480587 16858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:08.476650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:34163 (local address 127.14.41.67:49219)
0925 10:08:08.476807 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:08.476819 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:08.476881 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:08.477139 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:08.477141 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:08.477143 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:08.477284 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:08.477287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:08.478058 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:08.478059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:08.478983 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:08.478986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:08.479862 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:08.479864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:08.479975 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:08.479981 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:08:08.480079 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 10:08:08.480152 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":51,"thread_start_us":67,"threads_started":1}
I20260925 10:08:08.480654 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:08.476749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:39609 (local address 127.14.41.66:44873)
0925 10:08:08.476910 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:08.476913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:08.476959 (+ 46us) server_negotiation.cc:407] Connection header received
0925 10:08:08.477019 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:08.477020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:08.477031 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:08.477144 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:08.477317 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:08.477894 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:08.479124 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:08.479717 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:08.479848 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:08.479856 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:08.480052 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 10:08:08.480115 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
I20260925 10:08:08.480647 16539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dd8f755b9b4fde8ccdff1a421ea8f4" candidate_uuid: "63f3197999764c8cadbdd94ede65a75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d114b21c51064f0292a65a3aa3c55b34" is_pre_election: true
I20260925 10:08:08.480752 16539 raft_consensus.cc:2454] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63f3197999764c8cadbdd94ede65a75f in term 0.
I20260925 10:08:08.480928 16330 leader_election.cc:302] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63f3197999764c8cadbdd94ede65a75f, d114b21c51064f0292a65a3aa3c55b34; no voters:
I20260925 10:08:08.480957 16539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dd8f755b9b4fde8ccdff1a421ea8f4" candidate_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d114b21c51064f0292a65a3aa3c55b34" is_pre_election: true
I20260925 10:08:08.480968 16400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dd8f755b9b4fde8ccdff1a421ea8f4" candidate_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63f3197999764c8cadbdd94ede65a75f" is_pre_election: true
I20260925 10:08:08.481007 16810 raft_consensus.cc:2781] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:08.480979 16681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dd8f755b9b4fde8ccdff1a421ea8f4" candidate_uuid: "63f3197999764c8cadbdd94ede65a75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" is_pre_election: true
I20260925 10:08:08.481019 16539 raft_consensus.cc:2454] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3a8c6420224a1e98f2e272f0a0aa4c in term 0.
I20260925 10:08:08.481036 16400 raft_consensus.cc:2454] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af3a8c6420224a1e98f2e272f0a0aa4c in term 0.
I20260925 10:08:08.481050 16681 raft_consensus.cc:2454] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63f3197999764c8cadbdd94ede65a75f in term 0.
I20260925 10:08:08.481058 16810 raft_consensus.cc:482] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:08.481082 16810 raft_consensus.cc:3037] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:08.481194 16611 leader_election.cc:302] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af3a8c6420224a1e98f2e272f0a0aa4c, d114b21c51064f0292a65a3aa3c55b34; no voters:
I20260925 10:08:08.481323 16848 raft_consensus.cc:2781] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:08.481357 16848 raft_consensus.cc:482] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:08.481379 16848 raft_consensus.cc:3037] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:08.481920 16810 raft_consensus.cc:504] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:08.482076 16810 leader_election.cc:288] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [CANDIDATE]: Term 1 election: Requested vote from peers d114b21c51064f0292a65a3aa3c55b34 (127.14.41.66:44873), af3a8c6420224a1e98f2e272f0a0aa4c (127.14.41.67:34993)
I20260925 10:08:08.482118 16848 raft_consensus.cc:504] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:08.482229 16681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dd8f755b9b4fde8ccdff1a421ea8f4" candidate_uuid: "63f3197999764c8cadbdd94ede65a75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c"
I20260925 10:08:08.482245 16539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dd8f755b9b4fde8ccdff1a421ea8f4" candidate_uuid: "63f3197999764c8cadbdd94ede65a75f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d114b21c51064f0292a65a3aa3c55b34"
I20260925 10:08:08.482313 16539 raft_consensus.cc:3037] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:08.482333 16681 raft_consensus.cc:2379] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 63f3197999764c8cadbdd94ede65a75f in current term 1: Already voted for candidate af3a8c6420224a1e98f2e272f0a0aa4c in this term.
I20260925 10:08:08.482437 16400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dd8f755b9b4fde8ccdff1a421ea8f4" candidate_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63f3197999764c8cadbdd94ede65a75f"
I20260925 10:08:08.482546 16400 raft_consensus.cc:2379] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate af3a8c6420224a1e98f2e272f0a0aa4c in current term 1: Already voted for candidate 63f3197999764c8cadbdd94ede65a75f in this term.
I20260925 10:08:08.482586 16540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dd8f755b9b4fde8ccdff1a421ea8f4" candidate_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d114b21c51064f0292a65a3aa3c55b34"
I20260925 10:08:08.482266 16848 leader_election.cc:288] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c [CANDIDATE]: Term 1 election: Requested vote from peers 63f3197999764c8cadbdd94ede65a75f (127.14.41.65:34163), d114b21c51064f0292a65a3aa3c55b34 (127.14.41.66:44873)
I20260925 10:08:08.482788 16613 leader_election.cc:302] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af3a8c6420224a1e98f2e272f0a0aa4c; no voters: 63f3197999764c8cadbdd94ede65a75f, d114b21c51064f0292a65a3aa3c55b34
I20260925 10:08:08.482877 16848 raft_consensus.cc:2726] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 10:08:08.483180 16539 raft_consensus.cc:2454] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63f3197999764c8cadbdd94ede65a75f in term 1.
I20260925 10:08:08.483417 16330 leader_election.cc:302] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63f3197999764c8cadbdd94ede65a75f, d114b21c51064f0292a65a3aa3c55b34; no voters: af3a8c6420224a1e98f2e272f0a0aa4c
I20260925 10:08:08.483551 16810 raft_consensus.cc:2781] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:08.483615 16810 raft_consensus.cc:686] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [term 1 LEADER]: Becoming Leader. State: Replica: 63f3197999764c8cadbdd94ede65a75f, State: Running, Role: LEADER
I20260925 10:08:08.483678 16810 consensus_queue.cc:237] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } }
I20260925 10:08:08.484359 16241 catalog_manager.cc:5495] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f reported cstate change: term changed from 0 to 1, leader changed from <none> to 63f3197999764c8cadbdd94ede65a75f (127.14.41.65). New cstate: current_term: 1 leader_uuid: "63f3197999764c8cadbdd94ede65a75f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63f3197999764c8cadbdd94ede65a75f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 } health_report { overall_health: UNKNOWN } } }
I20260925 10:08:08.497128 16681 raft_consensus.cc:1261] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c [term 1 FOLLOWER]: Refusing update from remote peer 63f3197999764c8cadbdd94ede65a75f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:08.497157 16539 raft_consensus.cc:1261] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34 [term 1 FOLLOWER]: Refusing update from remote peer 63f3197999764c8cadbdd94ede65a75f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:08.497406 16813 consensus_queue.cc:1035] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3a8c6420224a1e98f2e272f0a0aa4c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:08.497534 16818 consensus_queue.cc:1035] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P 63f3197999764c8cadbdd94ede65a75f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d114b21c51064f0292a65a3aa3c55b34" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 10:08:08.502549 16811 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 10:08:10.021399 16240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:10.021697 16240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:10.022668 16240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a6f47c71e4d4988ab90685daa68c526: Sending DeleteTablet for 3 replicas of tablet b0dd8f755b9b4fde8ccdff1a421ea8f4
I20260925 10:08:10.023022 16380 tablet_service.cc:1514] Processing DeleteTablet for tablet b0dd8f755b9b4fde8ccdff1a421ea8f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:10 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:55622
I20260925 10:08:10.023079 16661 tablet_service.cc:1514] Processing DeleteTablet for tablet b0dd8f755b9b4fde8ccdff1a421ea8f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:10 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:50456
I20260925 10:08:10.023066 16520 tablet_service.cc:1514] Processing DeleteTablet for tablet b0dd8f755b9b4fde8ccdff1a421ea8f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:10 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:50486
I20260925 10:08:10.023151 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 16314
I20260925 10:08:10.023860 16884 tablet_replica.cc:330] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34: stopping tablet replica
I20260925 10:08:10.023962 16884 raft_consensus.cc:2229] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:10.024010 16884 raft_consensus.cc:2258] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:10.024262 16883 tablet_replica.cc:330] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c: stopping tablet replica
I20260925 10:08:10.024307 16883 raft_consensus.cc:2229] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:10.024334 16883 raft_consensus.cc:2258] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:10.024578 16883 ts_tablet_manager.cc:1905] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:10.025000 16884 ts_tablet_manager.cc:1905] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:10.026182 16883 ts_tablet_manager.cc:1918] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:08:10.026232 16883 log.cc:1196] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/b0dd8f755b9b4fde8ccdff1a421ea8f4
I20260925 10:08:10.026479 16883 ts_tablet_manager.cc:1939] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P af3a8c6420224a1e98f2e272f0a0aa4c: Deleting consensus metadata
I20260925 10:08:10.026698 16884 ts_tablet_manager.cc:1918] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:08:10.026750 16884 log.cc:1196] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/b0dd8f755b9b4fde8ccdff1a421ea8f4
I20260925 10:08:10.026821 16228 catalog_manager.cc:4841] TS af3a8c6420224a1e98f2e272f0a0aa4c (127.14.41.67:34993): tablet b0dd8f755b9b4fde8ccdff1a421ea8f4 (table test-table [id=ce302c4b849d4eafb00c1ed58fbeaa0a]) successfully deleted
I20260925 10:08:10.026988 16884 ts_tablet_manager.cc:1939] T b0dd8f755b9b4fde8ccdff1a421ea8f4 P d114b21c51064f0292a65a3aa3c55b34: Deleting consensus metadata
I20260925 10:08:10.027345 16226 catalog_manager.cc:4841] TS d114b21c51064f0292a65a3aa3c55b34 (127.14.41.66:44873): tablet b0dd8f755b9b4fde8ccdff1a421ea8f4 (table test-table [id=ce302c4b849d4eafb00c1ed58fbeaa0a]) successfully deleted
W20260925 10:08:10.030620 16228 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.41.65:34163 (error 108)
I20260925 10:08:10.030764 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 16455
I20260925 10:08:10.031726 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:10.031509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:34163 (local address 127.0.0.1:55658)
0925 10:08:10.031637 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:10.031690 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.65:34163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":82,"threads_started":1}
W20260925 10:08:10.031875 16228 catalog_manager.cc:4584] TS 63f3197999764c8cadbdd94ede65a75f (127.14.41.65:34163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0dd8f755b9b4fde8ccdff1a421ea8f4: Network error: Client connection negotiation failed: client connection to 127.14.41.65:34163: connect: Connection refused (error 111)
I20260925 10:08:10.037655 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 16595
I20260925 10:08:10.042768 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 16210
2026-09-25T10:08:10Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5842 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:08:10 dist-test-slave-s6tm krb5kdc[16890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:08:10 dist-test-slave-s6tm krb5kdc[16890](info): set up 2 sockets
Sep 25 10:08:10 dist-test-slave-s6tm krb5kdc[16890](info): commencing operation
krb5kdc: starting...
W20260925 10:08:11.101181 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:08:11 dist-test-slave-s6tm krb5kdc[16890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:08:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:08:11.227849 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:46343
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:46343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:11.317695 16903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:11.317953 16903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:11.317993 16903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:11.319164 16903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:11.319231 16903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:11.319257 16903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:11.319288 16903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:11.319310 16903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:08:11.321033 16903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:46343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:46343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:11.321316 16903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:11.321568 16903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:11.327034 16909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:11.327054 16911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:11.327072 16903 server_base.cc:1034] running on GCE node
W20260925 10:08:11.327034 16908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:11.327515 16903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:11.327821 16903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:11.329012 16903 hybrid_clock.cc:648] HybridClock initialized: now 1790330891328992 us; error 55 us; skew 500 ppm
Sep 25 10:08:11 dist-test-slave-s6tm krb5kdc[16890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330891, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:11.335400 16903 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:08:11.335897 16903 webserver.cc:460] Webserver started at http://127.14.41.126:40351/ using document root <none> and password file <none>
I20260925 10:08:11.336160 16903 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:11.336208 16903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:11.336310 16903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:11.337524 16903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "a88a4d5cfa834d129bcd1014c516a8da"
format_stamp: "Formatted at 2026-09-25 10:08:11 on dist-test-slave-s6tm"
server_key: "ecd488555208a751e7f78bdd6c957dab"
server_key_iv: "93d2abb0b8f4f8009ebe04f346e1f6c3"
server_key_version: "encryptionkey@0"
I20260925 10:08:11.337841 16903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "a88a4d5cfa834d129bcd1014c516a8da"
format_stamp: "Formatted at 2026-09-25 10:08:11 on dist-test-slave-s6tm"
server_key: "ecd488555208a751e7f78bdd6c957dab"
server_key_iv: "93d2abb0b8f4f8009ebe04f346e1f6c3"
server_key_version: "encryptionkey@0"
I20260925 10:08:11.339218 16903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:08:11.340024 16918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:11.340207 16903 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:11.340299 16903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "a88a4d5cfa834d129bcd1014c516a8da"
format_stamp: "Formatted at 2026-09-25 10:08:11 on dist-test-slave-s6tm"
server_key: "ecd488555208a751e7f78bdd6c957dab"
server_key_iv: "93d2abb0b8f4f8009ebe04f346e1f6c3"
server_key_version: "encryptionkey@0"
I20260925 10:08:11.340372 16903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:11.353232 16903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:11.355687 16903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:11.355813 16903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:11.360141 16903 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:46343
I20260925 10:08:11.360222 16970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:46343 every 8 connection(s)
I20260925 10:08:11.360762 16903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:11.361567 16971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:11.363427 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 16903
I20260925 10:08:11.363559 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:08:11.363745 14501 external_mini_cluster.cc:1428] Setting key c6fea27f78228d7bcddda1f746bf5781
I20260925 10:08:11.364280 16971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da: Bootstrap starting.
I20260925 10:08:11.365031 16971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:11.365365 16971 log.cc:824] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:11.366163 16971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da: No bootstrap required, opened a new log
I20260925 10:08:11.368613 16971 raft_consensus.cc:348] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a88a4d5cfa834d129bcd1014c516a8da" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46343 } }
I20260925 10:08:11.368811 16971 raft_consensus.cc:374] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:11.368847 16971 raft_consensus.cc:729] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a88a4d5cfa834d129bcd1014c516a8da, State: Initialized, Role: FOLLOWER
I20260925 10:08:11.368981 16971 consensus_queue.cc:260] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a88a4d5cfa834d129bcd1014c516a8da" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46343 } }
I20260925 10:08:11.369081 16971 raft_consensus.cc:388] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:08:11.369117 16971 raft_consensus.cc:482] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:11.369167 16971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:11.369796 16971 raft_consensus.cc:504] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a88a4d5cfa834d129bcd1014c516a8da" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46343 } }
I20260925 10:08:11.369944 16971 leader_election.cc:302] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a88a4d5cfa834d129bcd1014c516a8da; no voters:
I20260925 10:08:11.370144 16971 leader_election.cc:288] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:08:11.370173 16976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:11.370296 16976 raft_consensus.cc:686] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da [term 1 LEADER]: Becoming Leader. State: Replica: a88a4d5cfa834d129bcd1014c516a8da, State: Running, Role: LEADER
I20260925 10:08:11.370421 16971 sys_catalog.cc:564] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:11.370463 16976 consensus_queue.cc:237] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a88a4d5cfa834d129bcd1014c516a8da" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46343 } }
I20260925 10:08:11.370826 16976 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da [sys.catalog]: SysCatalogTable state changed. Reason: New leader a88a4d5cfa834d129bcd1014c516a8da. Latest consensus state: current_term: 1 leader_uuid: "a88a4d5cfa834d129bcd1014c516a8da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a88a4d5cfa834d129bcd1014c516a8da" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46343 } } }
I20260925 10:08:11.370959 16976 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:11.370836 16978 sys_catalog.cc:455] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a88a4d5cfa834d129bcd1014c516a8da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a88a4d5cfa834d129bcd1014c516a8da" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46343 } } }
I20260925 10:08:11.371122 16978 sys_catalog.cc:458] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:11.371223 16984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 10:08:11 dist-test-slave-s6tm krb5kdc[16890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:11.371770 16984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:11.374465 16984 catalog_manager.cc:1347] Generated new cluster ID: 15cabaf2c0144a6097e5062b8773c959
I20260925 10:08:11.374521 16984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:08:11.376078 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:11.364338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.14.41.126:46343)
0925 10:08:11.364541 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:11.364550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:11.364580 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:08:11.365230 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:11.365240 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:11.365311 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:11.365486 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:11.366130 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:11.367549 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:11.368483 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:11.369170 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:11.369283 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:11.372221 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:11.372257 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:11.372257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:11.372271 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:11.374317 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:11.374725 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:11.374728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:11.374729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:11.374853 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:11.375078 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:11.375079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:11.375079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:11.375410 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:11.375584 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:11.375656 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 10:08:11.375826 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260925 10:08:11.402661 16984 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:08:11.403272 16984 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:11.411286 16984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a88a4d5cfa834d129bcd1014c516a8da: Generated new TSK 0
I20260925 10:08:11.411602 16984 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
I20260925 10:08:11.443428 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46343
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:11.529318 16997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:11.529603 16997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:11.529645 16997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:11.530922 16997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:11.531009 16997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:11.531055 16997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:08:11.532707 16997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:11.533047 16997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:11.533293 16997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:11.538964 17005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:11.538970 17003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:11.538981 17002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:11.539224 16997 server_base.cc:1034] running on GCE node
I20260925 10:08:11.539527 16997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:11.539799 16997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:11.541029 16997 hybrid_clock.cc:648] HybridClock initialized: now 1790330891541014 us; error 30 us; skew 500 ppm
Sep 25 10:08:11 dist-test-slave-s6tm krb5kdc[16890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330891, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:11.544301 16997 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:08:11.544832 16997 webserver.cc:460] Webserver started at http://127.14.41.65:46817/ using document root <none> and password file <none>
I20260925 10:08:11.545120 16997 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:11.545202 16997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:11.545364 16997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:11.546548 16997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "26aa6606b06d4446a92f3dbbfd4b7b78"
format_stamp: "Formatted at 2026-09-25 10:08:11 on dist-test-slave-s6tm"
server_key: "dc0c5987216cdb4ecc17cc5cab178b5e"
server_key_iv: "bf448eac98e5a33d246707810c3c48ef"
server_key_version: "encryptionkey@0"
I20260925 10:08:11.547032 16997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "26aa6606b06d4446a92f3dbbfd4b7b78"
format_stamp: "Formatted at 2026-09-25 10:08:11 on dist-test-slave-s6tm"
server_key: "dc0c5987216cdb4ecc17cc5cab178b5e"
server_key_iv: "bf448eac98e5a33d246707810c3c48ef"
server_key_version: "encryptionkey@0"
I20260925 10:08:11.548676 16997 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260925 10:08:11.549556 17012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:11.550086 16997 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:11.550228 16997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "26aa6606b06d4446a92f3dbbfd4b7b78"
format_stamp: "Formatted at 2026-09-25 10:08:11 on dist-test-slave-s6tm"
server_key: "dc0c5987216cdb4ecc17cc5cab178b5e"
server_key_iv: "bf448eac98e5a33d246707810c3c48ef"
server_key_version: "encryptionkey@0"
I20260925 10:08:11.550338 16997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:11.565393 16997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:11.568429 16997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:11.568647 16997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:11.568944 16997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:11.569293 16997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:11.569350 16997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:11.569383 16997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:11.569401 16997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:11.575904 16997 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:33107
I20260925 10:08:11.575991 17125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:33107 every 8 connection(s)
I20260925 10:08:11.576323 16997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:08:11.579475 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 16997
I20260925 10:08:11.579584 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:08:11.579756 14501 external_mini_cluster.cc:1428] Setting key f62673ad0b46f164e63de676813da174
Sep 25 10:08:11 dist-test-slave-s6tm krb5kdc[16890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330891, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:11.587757 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:11.577129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:43329 (local address 127.14.41.126:46343)
0925 10:08:11.577219 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:11.577221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:11.577506 (+ 285us) server_negotiation.cc:407] Connection header received
0925 10:08:11.578137 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:11.578139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:11.578150 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:11.578216 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:11.579299 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:11.580203 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:11.581302 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:11.581524 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:11.581625 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:11.584317 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:11.584319 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:11.584320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:11.584329 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:11.586260 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:11.586704 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:11.586706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:11.586706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:11.586788 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:11.587039 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:11.587040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:11.587040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:11.587198 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:11.587276 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:11.587542 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 10:08:11.587638 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:08:11.587812 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:11.577085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46343 (local address 127.14.41.65:43329)
0925 10:08:11.577293 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:11.577367 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:11.578013 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:11.578288 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:11.578291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:11.578933 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:11.579149 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:11.579155 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:11.580358 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:11.580359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:11.581190 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:11.581192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:11.581664 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:11.581666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:11.581839 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:11.582438 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:11.582447 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:11.584168 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:11.586366 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:11.586374 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:11.586376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:11.586623 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:11.586863 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:11.586863 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:11.586864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:11.586973 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:11.587267 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:11.587272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:11.587403 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 10:08:11.587489 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 10:08:11.587624 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":107,"threads_started":1}
I20260925 10:08:11.588856 17126 heartbeater.cc:347] Connected to a master server at 127.14.41.126:46343
WARNING: no policy specified for kudu/127.14.41.66@KRBTEST.COM; defaulting to no policy
I20260925 10:08:11.588982 17126 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:11.589210 17126 heartbeater.cc:511] Master 127.14.41.126:46343 requested a full tablet report, sending...
I20260925 10:08:11.589910 16932 ts_manager.cc:194] Registered new tserver with Master: 26aa6606b06d4446a92f3dbbfd4b7b78 (127.14.41.65:33107)
I20260925 10:08:11.590694 16932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:43329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
I20260925 10:08:11.611609 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46343
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:11.698913 17131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:11.699131 17131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:11.699156 17131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:11.700368 17131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:11.700409 17131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:11.700435 17131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:08:11.701838 17131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:11.702037 17131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:11.702226 17131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:11.707592 17139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:11.707617 17136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:11.707661 17137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:11.708142 17131 server_base.cc:1034] running on GCE node
I20260925 10:08:11.708369 17131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:11.708634 17131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:11.709805 17131 hybrid_clock.cc:648] HybridClock initialized: now 1790330891709802 us; error 41 us; skew 500 ppm
Sep 25 10:08:11 dist-test-slave-s6tm krb5kdc[16890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330891, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:11.712826 17131 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:08:11.713569 17131 webserver.cc:460] Webserver started at http://127.14.41.66:33271/ using document root <none> and password file <none>
I20260925 10:08:11.713887 17131 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:11.713980 17131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:11.714126 17131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:11.715240 17131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "7fa8f73086944043bac61c6d823de188"
format_stamp: "Formatted at 2026-09-25 10:08:11 on dist-test-slave-s6tm"
server_key: "8ca8eda76d146e481a70bb9d9c93e16e"
server_key_iv: "14c5616aa13c1c0213dd1affe41aed32"
server_key_version: "encryptionkey@0"
I20260925 10:08:11.715622 17131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "7fa8f73086944043bac61c6d823de188"
format_stamp: "Formatted at 2026-09-25 10:08:11 on dist-test-slave-s6tm"
server_key: "8ca8eda76d146e481a70bb9d9c93e16e"
server_key_iv: "14c5616aa13c1c0213dd1affe41aed32"
server_key_version: "encryptionkey@0"
I20260925 10:08:11.719060 17131 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 10:08:11.720240 17146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:11.720466 17131 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:11.720647 17131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "7fa8f73086944043bac61c6d823de188"
format_stamp: "Formatted at 2026-09-25 10:08:11 on dist-test-slave-s6tm"
server_key: "8ca8eda76d146e481a70bb9d9c93e16e"
server_key_iv: "14c5616aa13c1c0213dd1affe41aed32"
server_key_version: "encryptionkey@0"
I20260925 10:08:11.720778 17131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:11.741271 17131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:11.743881 17131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:11.744088 17131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:11.744369 17131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:11.744704 17131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:11.744757 17131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:11.744792 17131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:11.744818 17131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:11.750726 17131 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:40833
I20260925 10:08:11.750834 17259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:40833 every 8 connection(s)
I20260925 10:08:11.751173 17131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:08:11.757220 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 17131
I20260925 10:08:11.757323 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:08:11.757515 14501 external_mini_cluster.cc:1428] Setting key a682c78d473e4462305a91b7b6b9cb44
Sep 25 10:08:11 dist-test-slave-s6tm krb5kdc[16890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330891, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:11.761822 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:11.752125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:44799 (local address 127.14.41.126:46343)
0925 10:08:11.752177 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:11.752179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:11.752450 (+ 271us) server_negotiation.cc:407] Connection header received
0925 10:08:11.753020 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:11.753022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:11.753035 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:11.753105 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:11.754148 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:11.754741 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:11.755863 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:11.756149 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:11.756237 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:11.758760 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:11.758762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:11.758763 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:11.758771 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:11.760310 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:11.760781 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:11.760783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:11.760783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:11.760847 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:11.761128 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:11.761129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:11.761130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:11.761257 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:11.761337 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:11.761642 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 10:08:11.761727 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:08:11.762029 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:11.752111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46343 (local address 127.14.41.66:44799)
0925 10:08:11.752316 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:11.752358 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:11.752909 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:11.753185 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:11.753190 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:11.753838 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:11.754040 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:11.754046 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:11.754944 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:11.754946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:11.755769 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:11.755771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:11.756249 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:11.756250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:11.756382 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:11.756897 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:11.756908 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:11.758655 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:11.760440 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:11.760447 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:11.760450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:11.760701 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:11.760936 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:11.760937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:11.760938 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:11.761056 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:11.761327 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:11.761335 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:11.761496 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 10:08:11.761605 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 10:08:11.761776 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260925 10:08:11.762645 17260 heartbeater.cc:347] Connected to a master server at 127.14.41.126:46343
I20260925 10:08:11.762748 17260 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:11.762964 17260 heartbeater.cc:511] Master 127.14.41.126:46343 requested a full tablet report, sending...
I20260925 10:08:11.763432 16932 ts_manager.cc:194] Registered new tserver with Master: 7fa8f73086944043bac61c6d823de188 (127.14.41.66:40833)
I20260925 10:08:11.764010 16932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:44799
WARNING: no policy specified for kudu/127.14.41.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
I20260925 10:08:11.787470 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46343
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:11.876890 17265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:11.877099 17265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:11.877122 17265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:11.878238 17265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:11.878275 17265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:11.878300 17265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:08:11.879673 17265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:11.879870 17265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:11.880131 17265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:11.885265 17273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:11.885286 17271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:11.885278 17270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:11.885489 17265 server_base.cc:1034] running on GCE node
I20260925 10:08:11.885691 17265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:11.885953 17265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:11.887111 17265 hybrid_clock.cc:648] HybridClock initialized: now 1790330891887100 us; error 33 us; skew 500 ppm
Sep 25 10:08:11 dist-test-slave-s6tm krb5kdc[16890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330891, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:11.889690 17265 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:08:11.890285 17265 webserver.cc:460] Webserver started at http://127.14.41.67:42809/ using document root <none> and password file <none>
I20260925 10:08:11.890547 17265 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:11.890616 17265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:11.890769 17265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:11.891856 17265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "1310b135d5ce4345898fefc2061c3771"
format_stamp: "Formatted at 2026-09-25 10:08:11 on dist-test-slave-s6tm"
server_key: "be8fc89b306015a310521332a0066939"
server_key_iv: "617edfd6b725ba3bf2040c9327f62569"
server_key_version: "encryptionkey@0"
I20260925 10:08:11.892287 17265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "1310b135d5ce4345898fefc2061c3771"
format_stamp: "Formatted at 2026-09-25 10:08:11 on dist-test-slave-s6tm"
server_key: "be8fc89b306015a310521332a0066939"
server_key_iv: "617edfd6b725ba3bf2040c9327f62569"
server_key_version: "encryptionkey@0"
I20260925 10:08:11.893731 17265 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:11.894580 17280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:11.894794 17265 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:11.894959 17265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "1310b135d5ce4345898fefc2061c3771"
format_stamp: "Formatted at 2026-09-25 10:08:11 on dist-test-slave-s6tm"
server_key: "be8fc89b306015a310521332a0066939"
server_key_iv: "617edfd6b725ba3bf2040c9327f62569"
server_key_version: "encryptionkey@0"
I20260925 10:08:11.895043 17265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:11.906452 17265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:11.908703 17265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:11.908962 17265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:11.909235 17265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:11.909565 17265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:11.909621 17265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:11.909664 17265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:11.909693 17265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:11.916205 17265 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:45157
I20260925 10:08:11.916249 17393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:45157 every 8 connection(s)
I20260925 10:08:11.916597 17265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:08:11.922765 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 17265
I20260925 10:08:11.922848 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:08:11.923050 14501 external_mini_cluster.cc:1428] Setting key 94a5e2b11a4a3f893a7839188a2c4313
Sep 25 10:08:11 dist-test-slave-s6tm krb5kdc[16890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330891, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:11.926517 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:11.917440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:38481 (local address 127.14.41.126:46343)
0925 10:08:11.917538 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:11.917541 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:11.917734 (+ 193us) server_negotiation.cc:407] Connection header received
0925 10:08:11.918302 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:11.918304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:11.918316 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:11.918375 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:11.919380 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:11.919900 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:11.920980 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:11.921178 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:11.921271 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:11.923585 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:11.923587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:11.923587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:11.923595 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:11.925193 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:11.925603 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:11.925603 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:11.925604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:11.925654 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:11.925899 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:11.925900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:11.925900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:11.926018 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:11.926070 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:11.926384 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 10:08:11.926451 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:08:11.926787 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:11.917394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46343 (local address 127.14.41.67:38481)
0925 10:08:11.917600 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:11.917648 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:11.918185 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:11.918460 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:11.918466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:11.919063 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:11.919263 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:11.919269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:11.920040 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:11.920041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:11.920871 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:11.920873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:11.921295 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:11.921296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:11.921423 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:11.921943 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:11.921954 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:11.923483 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:11.925260 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:11.925266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:11.925268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:11.925535 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:11.925715 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:11.925716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:11.925716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:11.925813 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:11.926078 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:11.926083 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:11.926260 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 10:08:11.926347 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 10:08:11.926509 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":96,"threads_started":1}
I20260925 10:08:11.927408 17394 heartbeater.cc:347] Connected to a master server at 127.14.41.126:46343
I20260925 10:08:11.927523 17394 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:11.927721 17394 heartbeater.cc:511] Master 127.14.41.126:46343 requested a full tablet report, sending...
I20260925 10:08:11.928238 16932 ts_manager.cc:194] Registered new tserver with Master: 1310b135d5ce4345898fefc2061c3771 (127.14.41.67:45157)
I20260925 10:08:11.928586 16932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:38481
I20260925 10:08:11.935338 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 10:08:11.947031 16974 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:11.945861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46938 (local address 127.14.41.126:46343)
0925 10:08:11.945943 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:11.945946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:11.946219 (+ 273us) server_negotiation.cc:407] Connection header received
0925 10:08:11.946330 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:11.946331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:11.946343 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:11.946386 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:11.946974 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46938: BlockingRecv error: recv got EOF from 127.0.0.1:46938 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 10:08:11.948256 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 16997
I20260925 10:08:11.952988 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 17131
I20260925 10:08:11.957628 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 17265
I20260925 10:08:11.962046 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 16903
2026-09-25T10:08:11Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1917 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:08:12 dist-test-slave-s6tm krb5kdc[17406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:08:12 dist-test-slave-s6tm krb5kdc[17406](info): set up 2 sockets
Sep 25 10:08:12 dist-test-slave-s6tm krb5kdc[17406](info): commencing operation
krb5kdc: starting...
W20260925 10:08:13.005231 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:08:13 dist-test-slave-s6tm krb5kdc[17406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:08:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:08:13.131740 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:37347
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:37347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:13.213694 17419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:13.213956 17419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:13.213999 17419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:13.215210 17419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:13.215286 17419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:13.215310 17419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:13.215332 17419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:13.215361 17419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:08:13.216991 17419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:37347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:37347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:13.217278 17419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:13.217520 17419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:13.222882 17425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:13.222954 17427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:13.223043 17424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:13.223232 17419 server_base.cc:1034] running on GCE node
I20260925 10:08:13.223619 17419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:13.223899 17419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:13.225111 17419 hybrid_clock.cc:648] HybridClock initialized: now 1790330893225091 us; error 40 us; skew 500 ppm
Sep 25 10:08:13 dist-test-slave-s6tm krb5kdc[17406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330893, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:13.228029 17419 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:08:13.228502 17419 webserver.cc:460] Webserver started at http://127.14.41.126:40501/ using document root <none> and password file <none>
I20260925 10:08:13.228893 17419 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:13.228971 17419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:13.229113 17419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:13.230284 17419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "fec1e1957097469d8abd1be48bc757b9"
format_stamp: "Formatted at 2026-09-25 10:08:13 on dist-test-slave-s6tm"
server_key: "8bcfc9d4173833b8dd37c4bd4e608723"
server_key_iv: "a088a333f6bc1b36347ee84201a63778"
server_key_version: "encryptionkey@0"
I20260925 10:08:13.230662 17419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "fec1e1957097469d8abd1be48bc757b9"
format_stamp: "Formatted at 2026-09-25 10:08:13 on dist-test-slave-s6tm"
server_key: "8bcfc9d4173833b8dd37c4bd4e608723"
server_key_iv: "a088a333f6bc1b36347ee84201a63778"
server_key_version: "encryptionkey@0"
I20260925 10:08:13.232153 17419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:08:13.233054 17434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:13.233284 17419 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:13.233398 17419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "fec1e1957097469d8abd1be48bc757b9"
format_stamp: "Formatted at 2026-09-25 10:08:13 on dist-test-slave-s6tm"
server_key: "8bcfc9d4173833b8dd37c4bd4e608723"
server_key_iv: "a088a333f6bc1b36347ee84201a63778"
server_key_version: "encryptionkey@0"
I20260925 10:08:13.233484 17419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:13.255460 17419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:13.257798 17419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:13.257989 17419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:13.262274 17419 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:37347
I20260925 10:08:13.262315 17486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:37347 every 8 connection(s)
I20260925 10:08:13.262810 17419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:13.263506 17487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:13.266084 17487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9: Bootstrap starting.
I20260925 10:08:13.266719 17487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:13.267081 17487 log.cc:824] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:13.267253 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 17419
I20260925 10:08:13.267369 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:08:13.267542 14501 external_mini_cluster.cc:1428] Setting key a1e5e3fe3d121992f71dee97644aad09
I20260925 10:08:13.267992 17487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9: No bootstrap required, opened a new log
I20260925 10:08:13.270324 17487 raft_consensus.cc:348] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec1e1957097469d8abd1be48bc757b9" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37347 } }
I20260925 10:08:13.270515 17487 raft_consensus.cc:374] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:13.270592 17487 raft_consensus.cc:729] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec1e1957097469d8abd1be48bc757b9, State: Initialized, Role: FOLLOWER
I20260925 10:08:13.270776 17487 consensus_queue.cc:260] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec1e1957097469d8abd1be48bc757b9" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37347 } }
I20260925 10:08:13.270851 17487 raft_consensus.cc:388] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:08:13.270874 17487 raft_consensus.cc:482] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:13.270903 17487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:13.271520 17487 raft_consensus.cc:504] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec1e1957097469d8abd1be48bc757b9" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37347 } }
I20260925 10:08:13.271651 17487 leader_election.cc:302] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fec1e1957097469d8abd1be48bc757b9; no voters:
I20260925 10:08:13.271858 17487 leader_election.cc:288] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:08:13.271893 17492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:13.272019 17492 raft_consensus.cc:686] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9 [term 1 LEADER]: Becoming Leader. State: Replica: fec1e1957097469d8abd1be48bc757b9, State: Running, Role: LEADER
I20260925 10:08:13.272171 17487 sys_catalog.cc:564] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:13.272166 17492 consensus_queue.cc:237] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec1e1957097469d8abd1be48bc757b9" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37347 } }
I20260925 10:08:13.272650 17494 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fec1e1957097469d8abd1be48bc757b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec1e1957097469d8abd1be48bc757b9" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37347 } } }
I20260925 10:08:13.272754 17494 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:13.273519 17498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:13.273000 17493 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fec1e1957097469d8abd1be48bc757b9. Latest consensus state: current_term: 1 leader_uuid: "fec1e1957097469d8abd1be48bc757b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec1e1957097469d8abd1be48bc757b9" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37347 } } }
I20260925 10:08:13.273762 17493 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:13.274071 17498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:13.276224 17498 catalog_manager.cc:1347] Generated new cluster ID: 06c29304d0c54557917f0875d32c961f
I20260925 10:08:13.276281 17498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:08:13 dist-test-slave-s6tm krb5kdc[17406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:13.280642 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:13.268073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41762 (local address 127.14.41.126:37347)
0925 10:08:13.268229 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:13.268236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:13.268262 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:08:13.270321 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:13.270330 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:13.270397 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:13.270613 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:13.271408 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:13.272670 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:13.273847 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:13.274455 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:13.274583 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:13.277239 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:13.277244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:13.277244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:13.277257 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:13.278993 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:13.279401 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:13.279405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:13.279405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:13.279509 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:13.279751 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:13.279752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:13.279752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:13.280067 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:13.280239 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:13.280310 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 10:08:13.280460 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
I20260925 10:08:13.281824 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:08:13.300029 17498 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:08:13.300807 17498 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:13.302471 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:13.297927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41778 (local address 127.14.41.126:37347)
0925 10:08:13.297977 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:13.297979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:13.298094 (+ 115us) server_negotiation.cc:407] Connection header received
0925 10:08:13.298149 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:13.298151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:13.298163 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:13.298245 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:13.298672 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:13.299818 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:13.300740 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:13.301053 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:13.301145 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:13.301940 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:13.301942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:13.301942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:13.302149 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:13.302166 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:13.302173 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:13.302240 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:13.302305 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 10:08:13.302355 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:08:13.306303 17498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fec1e1957097469d8abd1be48bc757b9: Generated new TSK 0
I20260925 10:08:13.306521 17498 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:08:13.314857 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 17419
2026-09-25T10:08:13Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1349 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:08:13 dist-test-slave-s6tm krb5kdc[17520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:08:13 dist-test-slave-s6tm krb5kdc[17520](info): set up 2 sockets
Sep 25 10:08:13 dist-test-slave-s6tm krb5kdc[17520](info): commencing operation
krb5kdc: starting...
W20260925 10:08:14.394133 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 10:08:14.625551 14501 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.231s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260925 10:08:14.792044 14501 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.166s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260925 10:08:14.975833 14501 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.184s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/test-user.keytab.
W20260925 10:08:15.595139 14501 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.619s user 0.000s sys 0.001s
Sep 25 10:08:15 dist-test-slave-s6tm krb5kdc[17520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:08:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
W20260925 10:08:16.628624 14501 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.14.41.126: real 1.010s user 0.000s sys 0.003s
I20260925 10:08:16.629096 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:43917
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:43917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:16.727633 17533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:16.727890 17533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:16.727916 17533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:16.729274 17533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:16.729879 17533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:16.729897 17533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:16.729911 17533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:16.729925 17533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:08:16.732024 17533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:43917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:43917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:16.732340 17533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:16.732556 17533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:16.738567 17538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:16.738565 17541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:16.738538 17539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:16.738787 17533 server_base.cc:1034] running on GCE node
I20260925 10:08:16.739162 17533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:16.739480 17533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:16.740669 17533 hybrid_clock.cc:648] HybridClock initialized: now 1790330896740619 us; error 66 us; skew 500 ppm
Sep 25 10:08:16 dist-test-slave-s6tm krb5kdc[17520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330896, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:16.747843 17533 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:08:16.748433 17533 webserver.cc:460] Webserver started at http://127.14.41.126:41911/ using document root <none> and password file <none>
I20260925 10:08:16.748694 17533 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:16.748770 17533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:16.748934 17533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:16.750108 17533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "d2e1e6a4e56b4beebbc720bb11a3648b"
format_stamp: "Formatted at 2026-09-25 10:08:16 on dist-test-slave-s6tm"
server_key: "16741cbbab10124139bbd688e09245a9"
server_key_iv: "4b502fc1de85a260c0e06c973ab51560"
server_key_version: "encryptionkey@0"
I20260925 10:08:16.750514 17533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "d2e1e6a4e56b4beebbc720bb11a3648b"
format_stamp: "Formatted at 2026-09-25 10:08:16 on dist-test-slave-s6tm"
server_key: "16741cbbab10124139bbd688e09245a9"
server_key_iv: "4b502fc1de85a260c0e06c973ab51560"
server_key_version: "encryptionkey@0"
I20260925 10:08:16.752341 17533 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260925 10:08:16.753386 17548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:16.753643 17533 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:16.753778 17533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "d2e1e6a4e56b4beebbc720bb11a3648b"
format_stamp: "Formatted at 2026-09-25 10:08:16 on dist-test-slave-s6tm"
server_key: "16741cbbab10124139bbd688e09245a9"
server_key_iv: "4b502fc1de85a260c0e06c973ab51560"
server_key_version: "encryptionkey@0"
I20260925 10:08:16.753875 17533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:16.798841 17533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:16.801656 17533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:16.801859 17533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:16.806617 17533 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:43917
I20260925 10:08:16.806691 17600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:43917 every 8 connection(s)
I20260925 10:08:16.807108 17533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:16.808454 17601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:16.811071 17601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b: Bootstrap starting.
I20260925 10:08:16.811718 17601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:16.812106 17601 log.cc:824] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:16.813064 17601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b: No bootstrap required, opened a new log
I20260925 10:08:16.815536 17601 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e1e6a4e56b4beebbc720bb11a3648b" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43917 } }
I20260925 10:08:16.815827 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 17533
I20260925 10:08:16.815837 17601 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:16.815901 17601 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2e1e6a4e56b4beebbc720bb11a3648b, State: Initialized, Role: FOLLOWER
I20260925 10:08:16.815951 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:08:16.816143 14501 external_mini_cluster.cc:1428] Setting key 3c5e3691813a386b1391fca2cab86f83
I20260925 10:08:16.816115 17601 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e1e6a4e56b4beebbc720bb11a3648b" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43917 } }
I20260925 10:08:16.816223 17601 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:08:16.816262 17601 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:16.816318 17601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b [term 0 FOLLOWER]: Advancing to term 1
Sep 25 10:08:16 dist-test-slave-s6tm krb5kdc[17520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:16.827211 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:16.816781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45234 (local address 127.14.41.126:43917)
0925 10:08:16.817084 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:16.817094 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:16.817124 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:08:16.817238 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:16.817244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:16.817298 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:16.817421 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:16.818170 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:16.819109 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:16.820200 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:16.820691 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:16.820781 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:16.823577 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:16.823582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:16.823582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:16.823594 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:16.825553 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:16.825969 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:16.825973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:16.825973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:16.826095 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:16.826346 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:16.826347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:16.826347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:16.826651 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:16.826767 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:16.826916 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 10:08:16.827031 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":196,"threads_started":1}
I20260925 10:08:16.886049 17601 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e1e6a4e56b4beebbc720bb11a3648b" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43917 } }
I20260925 10:08:16.886303 17601 leader_election.cc:302] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2e1e6a4e56b4beebbc720bb11a3648b; no voters:
I20260925 10:08:16.886585 17601 leader_election.cc:288] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:08:16.886685 17606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:16.886972 17601 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:16.887059 17606 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b [term 1 LEADER]: Becoming Leader. State: Replica: d2e1e6a4e56b4beebbc720bb11a3648b, State: Running, Role: LEADER
I20260925 10:08:16.887230 17606 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e1e6a4e56b4beebbc720bb11a3648b" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43917 } }
I20260925 10:08:16.887708 17607 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2e1e6a4e56b4beebbc720bb11a3648b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e1e6a4e56b4beebbc720bb11a3648b" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43917 } } }
I20260925 10:08:16.887730 17608 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2e1e6a4e56b4beebbc720bb11a3648b. Latest consensus state: current_term: 1 leader_uuid: "d2e1e6a4e56b4beebbc720bb11a3648b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e1e6a4e56b4beebbc720bb11a3648b" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43917 } } }
I20260925 10:08:16.887838 17607 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:16.887857 17608 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:16.888222 17615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:16.888916 17615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:16.891028 17615 catalog_manager.cc:1347] Generated new cluster ID: 45679d1875794c1c8a6603e067d75559
I20260925 10:08:16.891201 17615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:08:16.906271 17615 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:08:16.907063 17615 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:16.911684 17615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2e1e6a4e56b4beebbc720bb11a3648b: Generated new TSK 0
I20260925 10:08:16.912119 17615 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
W20260925 10:08:17.802739 14501 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.14.41.65: real 0.886s user 0.000s sys 0.003s
I20260925 10:08:17.803135 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:43917
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:17.888967 17627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:17.889236 17627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:17.889276 17627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:17.890481 17627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:17.890560 17627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:17.890607 17627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:08:17.892164 17627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:43917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:17.892463 17627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:17.892736 17627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:17.898677 17632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:17.898729 17633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:17.898677 17635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:17.899081 17627 server_base.cc:1034] running on GCE node
I20260925 10:08:17.899319 17627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:17.899639 17627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:17.900831 17627 hybrid_clock.cc:648] HybridClock initialized: now 1790330897900867 us; error 86 us; skew 500 ppm
Sep 25 10:08:17 dist-test-slave-s6tm krb5kdc[17520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330897, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:17.903681 17627 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:08:17.904220 17627 webserver.cc:460] Webserver started at http://127.14.41.65:33285/ using document root <none> and password file <none>
I20260925 10:08:17.904469 17627 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:17.904538 17627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:17.904703 17627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:17.905798 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "adb08a50880a467e87cd65be71ef3295"
format_stamp: "Formatted at 2026-09-25 10:08:17 on dist-test-slave-s6tm"
server_key: "ddd75bb4225afbe650d2d952605f7d1e"
server_key_iv: "630b77e451d26956a303e6c81d52d8d9"
server_key_version: "encryptionkey@0"
I20260925 10:08:17.906172 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "adb08a50880a467e87cd65be71ef3295"
format_stamp: "Formatted at 2026-09-25 10:08:17 on dist-test-slave-s6tm"
server_key: "ddd75bb4225afbe650d2d952605f7d1e"
server_key_iv: "630b77e451d26956a303e6c81d52d8d9"
server_key_version: "encryptionkey@0"
I20260925 10:08:17.907550 17627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:08:17.908386 17642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:17.908578 17627 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:17.908684 17627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "adb08a50880a467e87cd65be71ef3295"
format_stamp: "Formatted at 2026-09-25 10:08:17 on dist-test-slave-s6tm"
server_key: "ddd75bb4225afbe650d2d952605f7d1e"
server_key_iv: "630b77e451d26956a303e6c81d52d8d9"
server_key_version: "encryptionkey@0"
I20260925 10:08:17.908761 17627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:17.926395 17627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:17.929679 17627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:17.929837 17627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:17.930115 17627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:17.930455 17627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:17.930490 17627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:17.930542 17627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:17.930567 17627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:17.936630 17627 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:44541
I20260925 10:08:17.936720 17755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:44541 every 8 connection(s)
I20260925 10:08:17.937069 17627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:08:17.938839 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 17627
I20260925 10:08:17.938961 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:08:17.939183 14501 external_mini_cluster.cc:1428] Setting key f7fd719e0870d1cc7af8f3784a755734
Sep 25 10:08:17 dist-test-slave-s6tm krb5kdc[17520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330897, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.41.66@KRBTEST.COM; defaulting to no policy
I20260925 10:08:17.949494 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:17.937852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:56849 (local address 127.14.41.126:43917)
0925 10:08:17.938741 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:17.938743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:17.938752 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:08:17.938783 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:17.938784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:17.938799 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:17.938992 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:17.940240 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:17.941062 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:17.942679 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:17.942888 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:17.942957 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:17.945992 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:17.945993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:17.945994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:17.946002 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:17.947695 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:17.948269 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:17.948270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:17.948271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:17.948330 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:17.948715 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:17.948716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:17.948716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:17.948839 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:17.948900 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:17.949311 (+ 411us) server_negotiation.cc:299] Negotiation successful
0925 10:08:17.949415 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":77,"threads_started":1}
I20260925 10:08:17.949781 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:17.937832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43917 (local address 127.14.41.65:56849)
0925 10:08:17.938006 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:17.938045 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:17.938547 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:17.939142 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:17.939148 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:17.939873 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:17.940149 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:17.940154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:17.941163 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:17.941164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:17.942538 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:17.942541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:17.942982 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:17.942983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:17.943143 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:17.943735 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:17.943748 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:17.945865 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:17.947795 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:17.947805 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:17.947808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:17.948194 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:17.948461 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:17.948463 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:17.948463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:17.948629 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:17.948994 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:17.949001 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:17.949139 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 10:08:17.949235 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 10:08:17.949411 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260925 10:08:17.950556 17756 heartbeater.cc:347] Connected to a master server at 127.14.41.126:43917
I20260925 10:08:17.950666 17756 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:17.951037 17756 heartbeater.cc:511] Master 127.14.41.126:43917 requested a full tablet report, sending...
I20260925 10:08:17.951706 17565 ts_manager.cc:194] Registered new tserver with Master: adb08a50880a467e87cd65be71ef3295 (127.14.41.65:44541)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.66@KRBTEST.COM" created.
I20260925 10:08:17.952584 17565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:56849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
I20260925 10:08:17.972743 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:43917
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:18.056941 17762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:18.057204 17762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:18.057257 17762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:18.058605 17762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:18.058652 17762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:18.058681 17762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:08:18.060271 17762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:43917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:18.060546 17762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:18.060823 17762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:18.066318 17768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:18.066318 17767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:18.066524 17762 server_base.cc:1034] running on GCE node
W20260925 10:08:18.066318 17770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:18.066919 17762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:18.067209 17762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:18.068377 17762 hybrid_clock.cc:648] HybridClock initialized: now 1790330898068354 us; error 40 us; skew 500 ppm
Sep 25 10:08:18 dist-test-slave-s6tm krb5kdc[17520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330898, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:18.071341 17762 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:08:18.071828 17762 webserver.cc:460] Webserver started at http://127.14.41.66:43913/ using document root <none> and password file <none>
I20260925 10:08:18.072108 17762 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:18.072181 17762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:18.072343 17762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:18.073900 17762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "6aadf1720b3047829a4dd877e0923482"
format_stamp: "Formatted at 2026-09-25 10:08:18 on dist-test-slave-s6tm"
server_key: "ab66a3f860a352ded647d66f3b0ac6e3"
server_key_iv: "6b15e993833c9fa5642dd4a91de49647"
server_key_version: "encryptionkey@0"
I20260925 10:08:18.074371 17762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "6aadf1720b3047829a4dd877e0923482"
format_stamp: "Formatted at 2026-09-25 10:08:18 on dist-test-slave-s6tm"
server_key: "ab66a3f860a352ded647d66f3b0ac6e3"
server_key_iv: "6b15e993833c9fa5642dd4a91de49647"
server_key_version: "encryptionkey@0"
I20260925 10:08:18.076037 17762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260925 10:08:18.076936 17777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:18.077138 17762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:18.077244 17762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "6aadf1720b3047829a4dd877e0923482"
format_stamp: "Formatted at 2026-09-25 10:08:18 on dist-test-slave-s6tm"
server_key: "ab66a3f860a352ded647d66f3b0ac6e3"
server_key_iv: "6b15e993833c9fa5642dd4a91de49647"
server_key_version: "encryptionkey@0"
I20260925 10:08:18.077334 17762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:18.086184 17762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:18.088831 17762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:18.089053 17762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:18.089327 17762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:18.089656 17762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:18.089710 17762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:18.089752 17762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:18.089779 17762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:18.095630 17762 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:39675
I20260925 10:08:18.095671 17890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:39675 every 8 connection(s)
I20260925 10:08:18.096089 17762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:08:18.098595 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 17762
I20260925 10:08:18.098713 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:08:18.098922 14501 external_mini_cluster.cc:1428] Setting key 814c89d24a8978f4fc6dfc451120ecc9
Sep 25 10:08:18 dist-test-slave-s6tm krb5kdc[17520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330898, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.41.67@KRBTEST.COM; defaulting to no policy
I20260925 10:08:18.108382 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:18.096906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:56625 (local address 127.14.41.126:43917)
0925 10:08:18.096995 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:18.096997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:18.097141 (+ 144us) server_negotiation.cc:407] Connection header received
0925 10:08:18.097670 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:18.097671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:18.097687 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:18.097753 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:18.099046 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:18.099728 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:18.101346 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:18.101689 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:18.101787 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:18.104501 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:18.104503 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:18.104504 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:18.104514 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:18.106476 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:18.107015 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:18.107017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:18.107017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:18.107079 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:18.107384 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:18.107385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:18.107385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:18.107552 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:18.107631 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:18.108124 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 10:08:18.108234 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:08:18.108529 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:18.096858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43917 (local address 127.14.41.66:56625)
0925 10:08:18.097014 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:18.097050 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:18.097540 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:18.097832 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:18.097836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:18.098500 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:18.098822 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:18.098831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:18.099826 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:18.099827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:18.101244 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:18.101247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:18.101814 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:18.101816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:18.101931 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:18.102497 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:18.102508 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:18.104395 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:18.106617 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:18.106626 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:18.106629 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:18.106926 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:18.107158 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:18.107159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:18.107160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:18.107303 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:18.107638 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:18.107645 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:18.107880 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 10:08:18.108087 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 10:08:18.108246 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260925 10:08:18.109294 17891 heartbeater.cc:347] Connected to a master server at 127.14.41.126:43917
I20260925 10:08:18.109413 17891 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:18.109826 17891 heartbeater.cc:511] Master 127.14.41.126:43917 requested a full tablet report, sending...
I20260925 10:08:18.110340 17565 ts_manager.cc:194] Registered new tserver with Master: 6aadf1720b3047829a4dd877e0923482 (127.14.41.66:39675)
I20260925 10:08:18.110739 17565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:56625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
I20260925 10:08:18.128257 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:43917
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:18.213351 17896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:18.213580 17896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:18.213608 17896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:18.214792 17896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:18.214838 17896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:18.214867 17896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:08:18.216480 17896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:43917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:18.216703 17896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:18.216908 17896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:18.222596 17901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:18.222621 17902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:18.222708 17896 server_base.cc:1034] running on GCE node
W20260925 10:08:18.222596 17904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:18.223062 17896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:18.223380 17896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:18.224536 17896 hybrid_clock.cc:648] HybridClock initialized: now 1790330898224513 us; error 38 us; skew 500 ppm
Sep 25 10:08:18 dist-test-slave-s6tm krb5kdc[17520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330898, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:18.227281 17896 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:08:18.227744 17896 webserver.cc:460] Webserver started at http://127.14.41.67:33921/ using document root <none> and password file <none>
I20260925 10:08:18.228114 17896 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:18.228188 17896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:18.228350 17896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:18.229449 17896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "e3f6a943917b4a3d8037d890d25a483a"
format_stamp: "Formatted at 2026-09-25 10:08:18 on dist-test-slave-s6tm"
server_key: "9b8775bf7122901451072a691352d473"
server_key_iv: "49f06037e7833faf0f30bd643b9f9e9a"
server_key_version: "encryptionkey@0"
I20260925 10:08:18.229858 17896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "e3f6a943917b4a3d8037d890d25a483a"
format_stamp: "Formatted at 2026-09-25 10:08:18 on dist-test-slave-s6tm"
server_key: "9b8775bf7122901451072a691352d473"
server_key_iv: "49f06037e7833faf0f30bd643b9f9e9a"
server_key_version: "encryptionkey@0"
I20260925 10:08:18.231283 17896 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:08:18.232272 17911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:18.232458 17896 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:18.232569 17896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "e3f6a943917b4a3d8037d890d25a483a"
format_stamp: "Formatted at 2026-09-25 10:08:18 on dist-test-slave-s6tm"
server_key: "9b8775bf7122901451072a691352d473"
server_key_iv: "49f06037e7833faf0f30bd643b9f9e9a"
server_key_version: "encryptionkey@0"
I20260925 10:08:18.232646 17896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:18.257043 17896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:18.259742 17896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:18.259945 17896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:18.260205 17896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:18.260536 17896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:18.260592 17896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:18.260639 17896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:18.260664 17896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:18.266805 17896 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:38237
I20260925 10:08:18.266953 18024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:38237 every 8 connection(s)
I20260925 10:08:18.267305 17896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
Sep 25 10:08:18 dist-test-slave-s6tm krb5kdc[17520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330898, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:18.273991 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 17896
I20260925 10:08:18.274104 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:08:18.274331 14501 external_mini_cluster.cc:1428] Setting key b1ad5f955b08ba3e7b2d00433978fe59
I20260925 10:08:18.277832 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:18.268033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:46567 (local address 127.14.41.126:43917)
0925 10:08:18.268103 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:18.268106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:18.268351 (+ 245us) server_negotiation.cc:407] Connection header received
0925 10:08:18.268953 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:18.268954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:18.268964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:18.269019 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:18.269998 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:18.270531 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:18.271608 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:18.271801 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:18.271988 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:18.274604 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:18.274605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:18.274606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:18.274616 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:18.276501 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:18.276917 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:18.276918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:18.276919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:18.276979 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:18.277214 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:18.277215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:18.277215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:18.277355 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:18.277413 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:18.277677 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 10:08:18.277761 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:08:18.278069 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:18.268065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43917 (local address 127.14.41.67:46567)
0925 10:08:18.268218 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:18.268255 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:18.268782 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:18.269077 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:18.269080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:18.269691 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:18.269895 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:18.269900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:18.270629 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:18.270630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:18.271511 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:18.271514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:18.271884 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:18.271885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:18.272061 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:18.272661 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:18.272670 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:18.274477 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:18.276586 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:18.276594 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:18.276596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:18.276837 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:18.277046 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:18.277047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:18.277048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:18.277154 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:18.277418 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:18.277422 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:18.277549 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 10:08:18.277635 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 10:08:18.277766 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":88,"threads_started":1}
I20260925 10:08:18.278590 18025 heartbeater.cc:347] Connected to a master server at 127.14.41.126:43917
I20260925 10:08:18.278703 18025 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:18.278895 18025 heartbeater.cc:511] Master 127.14.41.126:43917 requested a full tablet report, sending...
I20260925 10:08:18.279297 17565 ts_manager.cc:194] Registered new tserver with Master: e3f6a943917b4a3d8037d890d25a483a (127.14.41.67:38237)
I20260925 10:08:18.279654 17565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:46567
I20260925 10:08:18.286576 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:08:18 dist-test-slave-s6tm krb5kdc[17520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330898, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 10:08:18 dist-test-slave-s6tm krb5kdc[17520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330898, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:18.314079 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:18.305387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.14.41.126:43917)
0925 10:08:18.305459 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:18.305461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:18.305471 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:08:18.305551 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:18.305551 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:18.305561 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:18.305617 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:18.306339 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:18.306900 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:18.307883 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:18.308149 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:18.308208 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:18.311096 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:18.311098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:18.311098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:18.311106 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:18.312879 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:18.313268 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:18.313270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:18.313270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:18.313337 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:18.313574 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:18.313575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:18.313576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:18.313749 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:18.313836 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:18.313929 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 10:08:18.313997 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 10:08:18.314401 17565 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45242
I20260925 10:08:18.315070 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 17627
I20260925 10:08:18.320010 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 17762
I20260925 10:08:18.325002 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 17896
I20260925 10:08:18.329334 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 17533
2026-09-25T10:08:18Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (5017 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:08:18 dist-test-slave-s6tm krb5kdc[18037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:08:18 dist-test-slave-s6tm krb5kdc[18037](info): set up 2 sockets
Sep 25 10:08:18 dist-test-slave-s6tm krb5kdc[18037](info): commencing operation
krb5kdc: starting...
W20260925 10:08:19.377368 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:08:19 dist-test-slave-s6tm krb5kdc[18037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:08:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:08:19.503798 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:42037
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:42037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:19.588121 18050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:19.588378 18050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:19.588419 18050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:19.589713 18050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:19.589797 18050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:19.589819 18050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:19.589856 18050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:19.589886 18050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:08:19.591490 18050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:42037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:42037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:19.591794 18050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:19.592105 18050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:19.597561 18058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:19.597605 18056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:19.597633 18055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:19.597636 18050 server_base.cc:1034] running on GCE node
I20260925 10:08:19.598049 18050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:19.598372 18050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:19.599577 18050 hybrid_clock.cc:648] HybridClock initialized: now 1790330899599581 us; error 66 us; skew 500 ppm
Sep 25 10:08:19 dist-test-slave-s6tm krb5kdc[18037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:19.602319 18050 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:08:19.602869 18050 webserver.cc:460] Webserver started at http://127.14.41.126:35407/ using document root <none> and password file <none>
I20260925 10:08:19.603113 18050 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:19.603180 18050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:19.603338 18050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:19.604543 18050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "b11e99ea285240ecad6c6bd19493b144"
format_stamp: "Formatted at 2026-09-25 10:08:19 on dist-test-slave-s6tm"
server_key: "dc73e3800a2fed97348d60b46c78dcdd"
server_key_iv: "c142527d23e22fc5b65d888e09c877a9"
server_key_version: "encryptionkey@0"
I20260925 10:08:19.604918 18050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "b11e99ea285240ecad6c6bd19493b144"
format_stamp: "Formatted at 2026-09-25 10:08:19 on dist-test-slave-s6tm"
server_key: "dc73e3800a2fed97348d60b46c78dcdd"
server_key_iv: "c142527d23e22fc5b65d888e09c877a9"
server_key_version: "encryptionkey@0"
I20260925 10:08:19.606323 18050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:08:19.607221 18065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:19.607462 18050 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:19.607576 18050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "b11e99ea285240ecad6c6bd19493b144"
format_stamp: "Formatted at 2026-09-25 10:08:19 on dist-test-slave-s6tm"
server_key: "dc73e3800a2fed97348d60b46c78dcdd"
server_key_iv: "c142527d23e22fc5b65d888e09c877a9"
server_key_version: "encryptionkey@0"
I20260925 10:08:19.607652 18050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:19.638281 18050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:19.640566 18050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:19.640751 18050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:19.645120 18050 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:42037
I20260925 10:08:19.645171 18117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:42037 every 8 connection(s)
I20260925 10:08:19.645730 18050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:19.646482 18118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:19.648988 18118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144: Bootstrap starting.
I20260925 10:08:19.649505 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 18050
I20260925 10:08:19.649617 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:08:19.649652 18118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:19.649806 14501 external_mini_cluster.cc:1428] Setting key f659c9aa2005c7bd1ea74a9e4652f6f7
I20260925 10:08:19.649986 18118 log.cc:824] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:19.650939 18118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144: No bootstrap required, opened a new log
I20260925 10:08:19.652931 18118 raft_consensus.cc:348] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11e99ea285240ecad6c6bd19493b144" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42037 } }
I20260925 10:08:19.653090 18118 raft_consensus.cc:374] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:19.653137 18118 raft_consensus.cc:729] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b11e99ea285240ecad6c6bd19493b144, State: Initialized, Role: FOLLOWER
I20260925 10:08:19.653306 18118 consensus_queue.cc:260] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11e99ea285240ecad6c6bd19493b144" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42037 } }
I20260925 10:08:19.653401 18118 raft_consensus.cc:388] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:08:19.653445 18118 raft_consensus.cc:482] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:19.653493 18118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:19.654038 18118 raft_consensus.cc:504] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11e99ea285240ecad6c6bd19493b144" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42037 } }
I20260925 10:08:19.654157 18118 leader_election.cc:302] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b11e99ea285240ecad6c6bd19493b144; no voters:
I20260925 10:08:19.654363 18118 leader_election.cc:288] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:08:19.654415 18123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:19.654623 18118 sys_catalog.cc:564] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:19.654672 18123 raft_consensus.cc:686] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144 [term 1 LEADER]: Becoming Leader. State: Replica: b11e99ea285240ecad6c6bd19493b144, State: Running, Role: LEADER
I20260925 10:08:19.654851 18123 consensus_queue.cc:237] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11e99ea285240ecad6c6bd19493b144" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42037 } }
I20260925 10:08:19.655280 18124 sys_catalog.cc:455] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b11e99ea285240ecad6c6bd19493b144" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11e99ea285240ecad6c6bd19493b144" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42037 } } }
I20260925 10:08:19.655396 18124 sys_catalog.cc:458] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:19.655584 18125 sys_catalog.cc:455] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b11e99ea285240ecad6c6bd19493b144. Latest consensus state: current_term: 1 leader_uuid: "b11e99ea285240ecad6c6bd19493b144" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11e99ea285240ecad6c6bd19493b144" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42037 } } }
I20260925 10:08:19.655638 18125 sys_catalog.cc:458] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:19.655841 18132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:19.656458 18132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 10:08:19 dist-test-slave-s6tm krb5kdc[18037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:19.661114 18132 catalog_manager.cc:1347] Generated new cluster ID: 4114316efe764e38bd95f6365d8c7dad
I20260925 10:08:19.661175 18132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:08:19.664433 18122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:19.650305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50112 (local address 127.14.41.126:42037)
0925 10:08:19.650477 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:19.650488 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:19.650527 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:08:19.651002 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:19.651011 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:19.651075 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:19.651230 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:19.651914 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:19.652844 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:19.653830 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:19.654378 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:19.655080 (+ 702us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:19.657715 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:19.657720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:19.657720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:19.657733 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:19.659657 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:19.660136 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:19.660140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:19.660140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:19.660242 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:19.660483 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:19.660484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:19.660484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:19.660799 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:19.660964 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:19.664148 (+ 3184us) server_negotiation.cc:299] Negotiation successful
0925 10:08:19.664268 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260925 10:08:19.682940 18132 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:08:19.683594 18132 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:19.689811 18132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b11e99ea285240ecad6c6bd19493b144: Generated new TSK 0
I20260925 10:08:19.690126 18132 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
I20260925 10:08:19.728520 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:42037
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:19.833375 18144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:19.833601 18144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:19.833628 18144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:19.835347 18144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:19.835407 18144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:19.835438 18144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:08:19.837003 18144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:42037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:19.837201 18144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:19.837385 18144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:19.842506 18152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:19.842506 18149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:19.842518 18150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:19.842963 18144 server_base.cc:1034] running on GCE node
I20260925 10:08:19.843230 18144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:19.843500 18144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:19.844684 18144 hybrid_clock.cc:648] HybridClock initialized: now 1790330899844643 us; error 35 us; skew 500 ppm
Sep 25 10:08:19 dist-test-slave-s6tm krb5kdc[18037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:19.847395 18144 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:08:19.847811 18144 webserver.cc:460] Webserver started at http://127.14.41.65:36955/ using document root <none> and password file <none>
I20260925 10:08:19.848064 18144 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:19.848112 18144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:19.848210 18144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:19.849236 18144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "efd4e02879f048bd83737d2e4a27aace"
format_stamp: "Formatted at 2026-09-25 10:08:19 on dist-test-slave-s6tm"
server_key: "c0f57a4a9cd127f1c2e95c819a4cff69"
server_key_iv: "7759454bf3cff4171bab91f60d17e962"
server_key_version: "encryptionkey@0"
I20260925 10:08:19.849566 18144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "efd4e02879f048bd83737d2e4a27aace"
format_stamp: "Formatted at 2026-09-25 10:08:19 on dist-test-slave-s6tm"
server_key: "c0f57a4a9cd127f1c2e95c819a4cff69"
server_key_iv: "7759454bf3cff4171bab91f60d17e962"
server_key_version: "encryptionkey@0"
I20260925 10:08:19.850929 18144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:08:19.851667 18159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:19.851821 18144 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:08:19.851907 18144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "efd4e02879f048bd83737d2e4a27aace"
format_stamp: "Formatted at 2026-09-25 10:08:19 on dist-test-slave-s6tm"
server_key: "c0f57a4a9cd127f1c2e95c819a4cff69"
server_key_iv: "7759454bf3cff4171bab91f60d17e962"
server_key_version: "encryptionkey@0"
I20260925 10:08:19.852015 18144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:19.862735 18144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:19.864779 18144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:19.864897 18144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:19.865091 18144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:19.865350 18144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:19.865388 18144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:19.865411 18144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:19.865423 18144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:19.871086 18144 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:37841
I20260925 10:08:19.871181 18272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:37841 every 8 connection(s)
I20260925 10:08:19.871533 18144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:08:19.874926 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 18144
I20260925 10:08:19.875048 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:08:19.875250 14501 external_mini_cluster.cc:1428] Setting key eadf5060b6fb0ddbe8c376abb066d543
Sep 25 10:08:19 dist-test-slave-s6tm krb5kdc[18037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:19.882270 18122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:19.872413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:44029 (local address 127.14.41.126:42037)
0925 10:08:19.872523 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:19.872525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:19.872659 (+ 134us) server_negotiation.cc:407] Connection header received
0925 10:08:19.873650 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:19.873651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:19.873661 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:19.873724 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:19.874333 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:19.874836 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:19.876426 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:19.876627 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:19.876683 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:19.879126 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:19.879128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:19.879128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:19.879135 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:19.880668 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:19.881241 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:19.881242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:19.881243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:19.881297 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:19.881599 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:19.881600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:19.881600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:19.881707 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:19.881760 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:19.882133 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 10:08:19.882201 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 10:08:19.882567 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:19.872341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:42037 (local address 127.14.41.65:44029)
0925 10:08:19.872527 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:19.872572 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:19.873557 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:19.873793 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:19.873796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:19.874063 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:19.874259 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:19.874264 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:19.875000 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:19.875001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:19.876336 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:19.876340 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:19.876712 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:19.876713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:19.876861 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:19.877510 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:19.877520 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:19.879027 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:19.880762 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:19.880772 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:19.880775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:19.881150 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:19.881365 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:19.881366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:19.881367 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:19.881527 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:19.881777 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:19.881785 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:19.881985 (+ 200us) client_negotiation.cc:769] Sending connection context
0925 10:08:19.882117 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 10:08:19.882300 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260925 10:08:19.883137 18273 heartbeater.cc:347] Connected to a master server at 127.14.41.126:42037
I20260925 10:08:19.883255 18273 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:19.883458 18273 heartbeater.cc:511] Master 127.14.41.126:42037 requested a full tablet report, sending...
I20260925 10:08:19.884084 18082 ts_manager.cc:194] Registered new tserver with Master: efd4e02879f048bd83737d2e4a27aace (127.14.41.65:37841)
WARNING: no policy specified for kudu/127.14.41.66@KRBTEST.COM; defaulting to no policy
I20260925 10:08:19.884809 18082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:44029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
I20260925 10:08:19.903666 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:42037
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:19.989750 18278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:19.989975 18278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:19.990002 18278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:19.991218 18278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:19.991271 18278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:19.991298 18278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:08:19.993038 18278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:42037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:19.993310 18278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:19.993574 18278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:19.999293 18286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:19.999297 18283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:19.999323 18284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:19.999698 18278 server_base.cc:1034] running on GCE node
I20260925 10:08:20.000070 18278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:20.000350 18278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:20.001529 18278 hybrid_clock.cc:648] HybridClock initialized: now 1790330900001511 us; error 40 us; skew 500 ppm
Sep 25 10:08:19 dist-test-slave-s6tm krb5kdc[18037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:20.004236 18278 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:08:20.004714 18278 webserver.cc:460] Webserver started at http://127.14.41.66:46393/ using document root <none> and password file <none>
I20260925 10:08:20.004997 18278 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:20.005070 18278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:20.005221 18278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:20.006330 18278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "81a735759ea642c19eab3e8eb2fc2f81"
format_stamp: "Formatted at 2026-09-25 10:08:20 on dist-test-slave-s6tm"
server_key: "b0eb00cb416a5ccf516528b14a0dca3c"
server_key_iv: "17ca8f89ef12dff111530c14b321360d"
server_key_version: "encryptionkey@0"
I20260925 10:08:20.006705 18278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "81a735759ea642c19eab3e8eb2fc2f81"
format_stamp: "Formatted at 2026-09-25 10:08:20 on dist-test-slave-s6tm"
server_key: "b0eb00cb416a5ccf516528b14a0dca3c"
server_key_iv: "17ca8f89ef12dff111530c14b321360d"
server_key_version: "encryptionkey@0"
I20260925 10:08:20.008230 18278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:08:20.009259 18293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:20.009445 18278 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:20.009542 18278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "81a735759ea642c19eab3e8eb2fc2f81"
format_stamp: "Formatted at 2026-09-25 10:08:20 on dist-test-slave-s6tm"
server_key: "b0eb00cb416a5ccf516528b14a0dca3c"
server_key_iv: "17ca8f89ef12dff111530c14b321360d"
server_key_version: "encryptionkey@0"
I20260925 10:08:20.009629 18278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:20.043165 18278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:20.045549 18278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:20.045674 18278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:20.045890 18278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:20.046156 18278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:20.046187 18278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:20.046211 18278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:20.046222 18278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:20.052979 18278 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:35845
I20260925 10:08:20.053018 18406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:35845 every 8 connection(s)
I20260925 10:08:20.053380 18278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:08:20.059337 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 18278
I20260925 10:08:20.059455 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:08:20.059656 14501 external_mini_cluster.cc:1428] Setting key 9ac12ae16b4076e57b4f029b6027e016
Sep 25 10:08:20 dist-test-slave-s6tm krb5kdc[18037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330899, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:20.064980 18122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:20.054175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:53611 (local address 127.14.41.126:42037)
0925 10:08:20.054234 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:20.054237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:20.054508 (+ 271us) server_negotiation.cc:407] Connection header received
0925 10:08:20.055687 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:20.055688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:20.055700 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:20.055821 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:20.056598 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:20.057174 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:20.058294 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:20.058494 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:20.058556 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:20.061586 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:20.061588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:20.061588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:20.061595 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:20.063284 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:20.063733 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:20.063734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:20.063735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:20.063805 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:20.064196 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:20.064197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:20.064197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:20.064342 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:20.064420 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:20.064781 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 10:08:20.064883 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 10:08:20.065200 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:20.054185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:42037 (local address 127.14.41.66:53611)
0925 10:08:20.054331 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:20.054372 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:20.055558 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:20.055907 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:20.055910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:20.056287 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:20.056493 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:20.056497 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:20.057378 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:20.057379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:20.058195 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:20.058196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:20.058576 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:20.058577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:20.058680 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:20.059248 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:20.059261 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:20.061471 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:20.063390 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:20.063397 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:20.063400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:20.063636 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:20.063987 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:20.063988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:20.063988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:20.064111 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:20.064424 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:20.064432 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:20.064618 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 10:08:20.064743 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 10:08:20.064941 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260925 10:08:20.065734 18407 heartbeater.cc:347] Connected to a master server at 127.14.41.126:42037
I20260925 10:08:20.065837 18407 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:20.066051 18407 heartbeater.cc:511] Master 127.14.41.126:42037 requested a full tablet report, sending...
I20260925 10:08:20.066493 18082 ts_manager.cc:194] Registered new tserver with Master: 81a735759ea642c19eab3e8eb2fc2f81 (127.14.41.66:35845)
I20260925 10:08:20.067023 18082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:53611
WARNING: no policy specified for kudu/127.14.41.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
I20260925 10:08:20.088201 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:42037
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:20.173621 18412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:20.173849 18412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:20.173877 18412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:20.175025 18412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:20.175069 18412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:20.175096 18412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:08:20.176649 18412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:42037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:20.176869 18412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:20.177048 18412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:20.182289 18418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:20.182332 18412 server_base.cc:1034] running on GCE node
W20260925 10:08:20.182248 18420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:20.182243 18417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:20.182751 18412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:20.183049 18412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:20.184195 18412 hybrid_clock.cc:648] HybridClock initialized: now 1790330900184183 us; error 27 us; skew 500 ppm
Sep 25 10:08:20 dist-test-slave-s6tm krb5kdc[18037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330900, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:20.187345 18412 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:08:20.187847 18412 webserver.cc:460] Webserver started at http://127.14.41.67:46291/ using document root <none> and password file <none>
I20260925 10:08:20.188133 18412 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:20.188210 18412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:20.188366 18412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:20.189563 18412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "23a5c87493cb4d0d895c2f142a760929"
format_stamp: "Formatted at 2026-09-25 10:08:20 on dist-test-slave-s6tm"
server_key: "eded40dc811c0e3f9d7e51ca9ff875e7"
server_key_iv: "52d7ca31400c504c55ca1cdabab3a027"
server_key_version: "encryptionkey@0"
I20260925 10:08:20.190033 18412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "23a5c87493cb4d0d895c2f142a760929"
format_stamp: "Formatted at 2026-09-25 10:08:20 on dist-test-slave-s6tm"
server_key: "eded40dc811c0e3f9d7e51ca9ff875e7"
server_key_iv: "52d7ca31400c504c55ca1cdabab3a027"
server_key_version: "encryptionkey@0"
I20260925 10:08:20.191602 18412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:08:20.192461 18427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:20.192667 18412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:08:20.192756 18412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "23a5c87493cb4d0d895c2f142a760929"
format_stamp: "Formatted at 2026-09-25 10:08:20 on dist-test-slave-s6tm"
server_key: "eded40dc811c0e3f9d7e51ca9ff875e7"
server_key_iv: "52d7ca31400c504c55ca1cdabab3a027"
server_key_version: "encryptionkey@0"
I20260925 10:08:20.192829 18412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:20.201252 18412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:20.203835 18412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:20.204062 18412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:20.204324 18412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:20.204679 18412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:20.204715 18412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:20.204737 18412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:20.204751 18412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:20.211006 18412 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:40213
I20260925 10:08:20.211124 18540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:40213 every 8 connection(s)
I20260925 10:08:20.211431 18412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:08:20.213779 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 18412
I20260925 10:08:20.213888 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:08:20.214071 14501 external_mini_cluster.cc:1428] Setting key c7c76af6ab362415b7547be0b5d25fcd
Sep 25 10:08:20 dist-test-slave-s6tm krb5kdc[18037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330900, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:20.222131 18122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:20.212198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:43425 (local address 127.14.41.126:42037)
0925 10:08:20.212288 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:20.212292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:20.212516 (+ 224us) server_negotiation.cc:407] Connection header received
0925 10:08:20.213640 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:20.213642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:20.213658 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:20.213832 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:20.214735 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:20.215431 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:20.216612 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:20.216843 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:20.216912 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:20.219227 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:20.219228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:20.219228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:20.219236 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:20.220675 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:20.221091 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:20.221092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:20.221092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:20.221152 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:20.221407 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:20.221409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:20.221409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:20.221549 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:20.221617 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:20.221988 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 10:08:20.222062 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:08:20.222316 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:20.212175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:42037 (local address 127.14.41.67:43425)
0925 10:08:20.212372 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:20.212424 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:20.213515 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:20.213921 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:20.213926 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:20.214297 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:20.214640 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:20.214647 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:20.215528 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:20.215529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:20.216512 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:20.216514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:20.216938 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:20.216939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:20.217050 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:20.217612 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:20.217622 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:20.219131 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:20.220753 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:20.220760 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:20.220761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:20.220998 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:20.221220 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:20.221221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:20.221221 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:20.221322 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:20.221666 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:20.221673 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:20.221828 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 10:08:20.221936 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:08:20.222067 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
I20260925 10:08:20.222878 18541 heartbeater.cc:347] Connected to a master server at 127.14.41.126:42037
I20260925 10:08:20.223018 18541 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:20.223243 18541 heartbeater.cc:511] Master 127.14.41.126:42037 requested a full tablet report, sending...
I20260925 10:08:20.223775 18082 ts_manager.cc:194] Registered new tserver with Master: 23a5c87493cb4d0d895c2f142a760929 (127.14.41.67:40213)
I20260925 10:08:20.224293 18082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:43425
I20260925 10:08:20.226183 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:08:20 dist-test-slave-s6tm krb5kdc[18037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:08:20 dist-test-slave-s6tm krb5kdc[18037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.65@KRBTEST.COM
I20260925 10:08:20.250974 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:20.242573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.14.41.65:37841)
0925 10:08:20.242733 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:20.242740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:20.242842 (+ 102us) server_negotiation.cc:407] Connection header received
0925 10:08:20.242872 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:20.242875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:20.242890 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:20.242963 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:20.243549 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:20.244468 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:20.245213 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:20.245554 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:20.245615 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:20.247901 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:20.247909 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:20.247910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:20.247917 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:20.249594 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:20.249932 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:20.249935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:20.249935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:20.250024 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:20.250205 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:20.250206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:20.250206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:20.250611 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:20.250735 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:20.250814 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 10:08:20.250885 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
Sep 25 10:08:20 dist-test-slave-s6tm krb5kdc[18037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:20.259869 18122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:20.252450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50114 (local address 127.14.41.126:42037)
0925 10:08:20.252520 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:20.252522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:20.252588 (+ 66us) server_negotiation.cc:407] Connection header received
0925 10:08:20.252679 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:20.252680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:20.252692 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:20.252758 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:20.253220 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:20.253720 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:20.254497 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:20.254756 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:20.254838 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:20.257243 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:20.257245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:20.257246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:20.257254 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:20.258668 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:20.259023 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:20.259024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:20.259025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:20.259088 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:20.259291 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:20.259292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:20.259292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:20.259452 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:20.259559 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:20.259670 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 10:08:20.259760 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 10:08:20.260221 18082 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50114
I20260925 10:08:20.260632 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 18144
I20260925 10:08:20.266268 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 18278
I20260925 10:08:20.271129 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 18412
I20260925 10:08:20.275972 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 18050
2026-09-25T10:08:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1946 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:08:20 dist-test-slave-s6tm krb5kdc[18553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:08:20 dist-test-slave-s6tm krb5kdc[18553](info): set up 2 sockets
Sep 25 10:08:20 dist-test-slave-s6tm krb5kdc[18553](info): commencing operation
krb5kdc: starting...
W20260925 10:08:21.373469 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:08:21 dist-test-slave-s6tm krb5kdc[18553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:08:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:08:21.505187 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:42259
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:42259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:21.589167 18566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:21.589440 18566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:21.589480 18566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:21.590667 18566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:21.590750 18566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:21.590775 18566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:21.590795 18566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:21.590832 18566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:08:21.592455 18566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:42259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:42259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:21.592767 18566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:21.593014 18566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:21.598734 18574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:21.598734 18572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:21.598747 18571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:21.599063 18566 server_base.cc:1034] running on GCE node
I20260925 10:08:21.599337 18566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:21.599661 18566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:21.600869 18566 hybrid_clock.cc:648] HybridClock initialized: now 1790330901600805 us; error 78 us; skew 500 ppm
Sep 25 10:08:21 dist-test-slave-s6tm krb5kdc[18553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330901, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:21.603835 18566 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:08:21.604223 18566 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:21.604272 18566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:21.604367 18566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:21.605417 18566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "62036e9eae7a4f93b1eb263a22e5f278"
format_stamp: "Formatted at 2026-09-25 10:08:21 on dist-test-slave-s6tm"
server_key: "d482b5af3919945a307ef8b76cb56732"
server_key_iv: "d3717ca85b58665de683461ce9c84979"
server_key_version: "encryptionkey@0"
I20260925 10:08:21.605722 18566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "62036e9eae7a4f93b1eb263a22e5f278"
format_stamp: "Formatted at 2026-09-25 10:08:21 on dist-test-slave-s6tm"
server_key: "d482b5af3919945a307ef8b76cb56732"
server_key_iv: "d3717ca85b58665de683461ce9c84979"
server_key_version: "encryptionkey@0"
I20260925 10:08:21.607057 18566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:08:21.608059 18580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:21.608243 18566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:21.608381 18566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "62036e9eae7a4f93b1eb263a22e5f278"
format_stamp: "Formatted at 2026-09-25 10:08:21 on dist-test-slave-s6tm"
server_key: "d482b5af3919945a307ef8b76cb56732"
server_key_iv: "d3717ca85b58665de683461ce9c84979"
server_key_version: "encryptionkey@0"
I20260925 10:08:21.608441 18566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:21.617441 18566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:21.619763 18566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:21.619887 18566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:21.623871 18566 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:42259
I20260925 10:08:21.624011 18632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:42259 every 8 connection(s)
I20260925 10:08:21.624298 18566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:21.625269 18633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:21.627637 18633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278: Bootstrap starting.
I20260925 10:08:21.628315 18633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:21.628633 18633 log.cc:824] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:21.629390 18633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278: No bootstrap required, opened a new log
I20260925 10:08:21.630558 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 18566
I20260925 10:08:21.630650 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:08:21.630884 14501 external_mini_cluster.cc:1428] Setting key fea89f851333be701a54d29d469f4d18
I20260925 10:08:21.631238 18633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62036e9eae7a4f93b1eb263a22e5f278" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42259 } }
I20260925 10:08:21.631479 18633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:21.631543 18633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62036e9eae7a4f93b1eb263a22e5f278, State: Initialized, Role: FOLLOWER
I20260925 10:08:21.631711 18633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62036e9eae7a4f93b1eb263a22e5f278" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42259 } }
I20260925 10:08:21.631793 18633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:08:21.631815 18633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:21.631886 18633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:21.632504 18633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62036e9eae7a4f93b1eb263a22e5f278" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42259 } }
I20260925 10:08:21.632622 18633 leader_election.cc:302] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62036e9eae7a4f93b1eb263a22e5f278; no voters:
I20260925 10:08:21.632815 18633 leader_election.cc:288] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:08:21.632875 18638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:21.633097 18633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:21.633118 18638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278 [term 1 LEADER]: Becoming Leader. State: Replica: 62036e9eae7a4f93b1eb263a22e5f278, State: Running, Role: LEADER
I20260925 10:08:21.634521 18638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62036e9eae7a4f93b1eb263a22e5f278" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42259 } }
I20260925 10:08:21.635138 18639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62036e9eae7a4f93b1eb263a22e5f278" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62036e9eae7a4f93b1eb263a22e5f278" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42259 } } }
I20260925 10:08:21.635270 18640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62036e9eae7a4f93b1eb263a22e5f278. Latest consensus state: current_term: 1 leader_uuid: "62036e9eae7a4f93b1eb263a22e5f278" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62036e9eae7a4f93b1eb263a22e5f278" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42259 } } }
W20260925 10:08:21.635681 18652 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 10:08:21.636116 18639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:21.636128 18640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278 [sys.catalog]: This master's current role is: LEADER
W20260925 10:08:21.636140 18652 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 10:08:21 dist-test-slave-s6tm krb5kdc[18553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:21.640087 18655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:21.640317 18655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:21.642283 18655 catalog_manager.cc:1347] Generated new cluster ID: 87286bd60d664deca256ecf9c29c9c41
I20260925 10:08:21.642333 18655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:08:21.642699 18637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:21.631495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.14.41.126:42259)
0925 10:08:21.631694 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:21.631704 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:21.631736 (+ 32us) server_negotiation.cc:407] Connection header received
0925 10:08:21.632232 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:21.632239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:21.632292 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:21.632419 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:21.633264 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:21.634371 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:21.635487 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:21.636148 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:21.636261 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:21.638767 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:21.638772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:21.638772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:21.638793 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:21.640837 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:21.641254 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:21.641258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:21.641259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:21.641392 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:21.641646 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:21.641647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:21.641648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:21.641986 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:21.642209 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:21.642279 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 10:08:21.642540 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
I20260925 10:08:21.663970 18655 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:08:21.664561 18655 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:21.676875 18655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278: Generated new TSK 0
I20260925 10:08:21.677138 18655 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
I20260925 10:08:21.707110 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:42259
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:21.796425 18659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:21.796674 18659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:21.796710 18659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:21.797901 18659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:21.797977 18659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:21.798026 18659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:08:21.799651 18659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:42259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:21.799997 18659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:21.800247 18659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:21.805471 18665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:21.805500 18667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:21.805490 18664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:21.805876 18659 server_base.cc:1034] running on GCE node
I20260925 10:08:21.806129 18659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:21.806393 18659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:21.807546 18659 hybrid_clock.cc:648] HybridClock initialized: now 1790330901807534 us; error 29 us; skew 500 ppm
Sep 25 10:08:21 dist-test-slave-s6tm krb5kdc[18553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330901, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:21.816284 18659 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:08:21.816712 18659 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:21.816797 18659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:21.816965 18659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:21.818183 18659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "4d670eecb9ca4447ae8d76637fe76841"
format_stamp: "Formatted at 2026-09-25 10:08:21 on dist-test-slave-s6tm"
server_key: "74d1c57f60e555f98d3554ec284037c9"
server_key_iv: "d9262e9a699da65978dd6c8b1a50a427"
server_key_version: "encryptionkey@0"
I20260925 10:08:21.818609 18659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "4d670eecb9ca4447ae8d76637fe76841"
format_stamp: "Formatted at 2026-09-25 10:08:21 on dist-test-slave-s6tm"
server_key: "74d1c57f60e555f98d3554ec284037c9"
server_key_iv: "d9262e9a699da65978dd6c8b1a50a427"
server_key_version: "encryptionkey@0"
I20260925 10:08:21.820350 18659 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 10:08:21.821314 18673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:21.821574 18659 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:21.821703 18659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "4d670eecb9ca4447ae8d76637fe76841"
format_stamp: "Formatted at 2026-09-25 10:08:21 on dist-test-slave-s6tm"
server_key: "74d1c57f60e555f98d3554ec284037c9"
server_key_iv: "d9262e9a699da65978dd6c8b1a50a427"
server_key_version: "encryptionkey@0"
I20260925 10:08:21.821928 18659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:21.855895 18659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:21.858242 18659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:21.858439 18659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:21.858713 18659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:21.859054 18659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:21.859112 18659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:21.859148 18659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:21.859196 18659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:21.865243 18659 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:44707
I20260925 10:08:21.865286 18786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:44707 every 8 connection(s)
I20260925 10:08:21.865583 18659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
Sep 25 10:08:21 dist-test-slave-s6tm krb5kdc[18553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330901, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:21.873435 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 18659
I20260925 10:08:21.873559 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:08:21.873770 14501 external_mini_cluster.cc:1428] Setting key 5efbef554acf7fd3a71f7ec6026a1de3
I20260925 10:08:21.876024 18637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:21.866332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:35553 (local address 127.14.41.126:42259)
0925 10:08:21.866405 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:21.866407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:21.866681 (+ 274us) server_negotiation.cc:407] Connection header received
0925 10:08:21.867160 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:21.867162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:21.867176 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:21.867239 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:21.868305 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:21.868833 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:21.869847 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:21.870061 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:21.870126 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:21.872677 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:21.872678 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:21.872679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:21.872689 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:21.874456 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:21.874956 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:21.874958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:21.874959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:21.875026 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:21.875292 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:21.875294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:21.875294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:21.875430 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:21.875513 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:21.875796 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 10:08:21.875887 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:08:21.876289 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:21.866342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:42259 (local address 127.14.41.65:35553)
0925 10:08:21.866540 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:21.866579 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:21.867063 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:21.867317 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:21.867320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:21.867590 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:21.868196 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:21.868202 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:21.868915 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:21.868915 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:21.869753 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:21.869755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:21.870149 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:21.870150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:21.870259 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:21.870833 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:21.870849 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:21.872561 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:21.874559 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:21.874567 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:21.874570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:21.874854 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:21.875097 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:21.875098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:21.875098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:21.875218 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:21.875504 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:21.875511 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:21.875656 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 10:08:21.875773 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 10:08:21.875939 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":88,"threads_started":1}
I20260925 10:08:21.876819 18787 heartbeater.cc:347] Connected to a master server at 127.14.41.126:42259
I20260925 10:08:21.876925 18787 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:21.877079 18787 heartbeater.cc:511] Master 127.14.41.126:42259 requested a full tablet report, sending...
I20260925 10:08:21.877632 18597 ts_manager.cc:194] Registered new tserver with Master: 4d670eecb9ca4447ae8d76637fe76841 (127.14.41.65:44707)
I20260925 10:08:21.878289 18597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:35553
WARNING: no policy specified for kudu/127.14.41.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
I20260925 10:08:21.903290 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:42259
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:21.989399 18792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:21.989629 18792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:21.989655 18792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:21.990823 18792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:21.990866 18792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:21.990892 18792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:08:21.992419 18792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:42259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:21.992628 18792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:21.992834 18792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:21.998718 18797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:21.998719 18798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:21.999076 18792 server_base.cc:1034] running on GCE node
W20260925 10:08:21.998718 18800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:21.999459 18792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:21.999747 18792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:22.000962 18792 hybrid_clock.cc:648] HybridClock initialized: now 1790330902000931 us; error 40 us; skew 500 ppm
Sep 25 10:08:21 dist-test-slave-s6tm krb5kdc[18553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330901, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:22.004071 18792 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:08:22.004560 18792 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:22.004642 18792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:22.004813 18792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:22.006042 18792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "fb28bf62c00c4decb85310fa4a7a40ab"
format_stamp: "Formatted at 2026-09-25 10:08:22 on dist-test-slave-s6tm"
server_key: "d97cd71f37e33aa0d7dc91f23ce0662f"
server_key_iv: "bb4d5113aca6906295f74317df9fb85e"
server_key_version: "encryptionkey@0"
I20260925 10:08:22.006451 18792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "fb28bf62c00c4decb85310fa4a7a40ab"
format_stamp: "Formatted at 2026-09-25 10:08:22 on dist-test-slave-s6tm"
server_key: "d97cd71f37e33aa0d7dc91f23ce0662f"
server_key_iv: "bb4d5113aca6906295f74317df9fb85e"
server_key_version: "encryptionkey@0"
I20260925 10:08:22.008114 18792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:08:22.009167 18806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:22.009418 18792 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:22.009544 18792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "fb28bf62c00c4decb85310fa4a7a40ab"
format_stamp: "Formatted at 2026-09-25 10:08:22 on dist-test-slave-s6tm"
server_key: "d97cd71f37e33aa0d7dc91f23ce0662f"
server_key_iv: "bb4d5113aca6906295f74317df9fb85e"
server_key_version: "encryptionkey@0"
I20260925 10:08:22.009635 18792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:22.036365 18792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:22.039075 18792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:22.039270 18792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:22.039592 18792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:22.039973 18792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:22.040032 18792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:22.040069 18792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:22.040097 18792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:22.046609 18792 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:35623
I20260925 10:08:22.046649 18919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:35623 every 8 connection(s)
I20260925 10:08:22.046953 18792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:08:22.049194 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 18792
I20260925 10:08:22.049327 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:08:22.049568 14501 external_mini_cluster.cc:1428] Setting key f356fd351dc9108afdf6bbd816ca4c05
Sep 25 10:08:22 dist-test-slave-s6tm krb5kdc[18553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330901, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.41.67@KRBTEST.COM; defaulting to no policy
I20260925 10:08:22.058439 18637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.047769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:50347 (local address 127.14.41.126:42259)
0925 10:08:22.047865 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:22.047868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:22.048030 (+ 162us) server_negotiation.cc:407] Connection header received
0925 10:08:22.048523 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:22.048524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:22.048535 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:22.048592 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:22.049771 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.050550 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.051856 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.052146 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.052247 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:22.054982 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:22.054984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:22.054984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:22.054993 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:22.056711 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:22.057262 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:22.057264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:22.057264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:22.057340 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:22.057621 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:22.057622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:22.057623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:22.057753 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:22.057905 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:22.058224 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 10:08:22.058325 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:08:22.058650 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.047718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:42259 (local address 127.14.41.66:50347)
0925 10:08:22.047880 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:22.047919 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:22.048425 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:22.048651 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:22.048655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:22.048963 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:22.049629 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:22.049637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.050681 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.050682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:22.051726 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:22.051730 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.052260 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.052262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:22.052413 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:22.053037 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:22.053046 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:22.054848 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:22.056909 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:22.056919 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:22.056922 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:22.057175 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:22.057413 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:22.057414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:22.057414 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:22.057538 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:22.057895 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:22.057903 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:22.058073 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 10:08:22.058190 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 10:08:22.058393 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":96,"threads_started":1}
I20260925 10:08:22.059366 18920 heartbeater.cc:347] Connected to a master server at 127.14.41.126:42259
I20260925 10:08:22.059491 18920 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:22.059631 18920 heartbeater.cc:511] Master 127.14.41.126:42259 requested a full tablet report, sending...
I20260925 10:08:22.060149 18597 ts_manager.cc:194] Registered new tserver with Master: fb28bf62c00c4decb85310fa4a7a40ab (127.14.41.66:35623)
I20260925 10:08:22.060559 18597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:50347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
I20260925 10:08:22.079162 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:42259
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:22.163230 18925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:22.163507 18925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:22.163549 18925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:22.164870 18925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:22.164948 18925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:22.164995 18925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:08:22.166605 18925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:42259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:22.166909 18925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:22.167160 18925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:08:22.172478 18925 server_base.cc:1034] running on GCE node
W20260925 10:08:22.172461 18931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:22.172461 18930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:22.172461 18933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:22.173012 18925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:22.173348 18925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:22.174522 18925 hybrid_clock.cc:648] HybridClock initialized: now 1790330902174500 us; error 58 us; skew 500 ppm
Sep 25 10:08:22 dist-test-slave-s6tm krb5kdc[18553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330902, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:22.177634 18925 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:08:22.178013 18925 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:22.178066 18925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:22.178160 18925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:22.179222 18925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "2fb30134a8e0447fbe5c2e2bfa9ea856"
format_stamp: "Formatted at 2026-09-25 10:08:22 on dist-test-slave-s6tm"
server_key: "0f79d1b98e942427d20280d7d9c54162"
server_key_iv: "1cbcb7cf5a9c558cca3007ccea8754c9"
server_key_version: "encryptionkey@0"
I20260925 10:08:22.179540 18925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "2fb30134a8e0447fbe5c2e2bfa9ea856"
format_stamp: "Formatted at 2026-09-25 10:08:22 on dist-test-slave-s6tm"
server_key: "0f79d1b98e942427d20280d7d9c54162"
server_key_iv: "1cbcb7cf5a9c558cca3007ccea8754c9"
server_key_version: "encryptionkey@0"
I20260925 10:08:22.181020 18925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:08:22.181789 18939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:22.182060 18925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:22.182157 18925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "2fb30134a8e0447fbe5c2e2bfa9ea856"
format_stamp: "Formatted at 2026-09-25 10:08:22 on dist-test-slave-s6tm"
server_key: "0f79d1b98e942427d20280d7d9c54162"
server_key_iv: "1cbcb7cf5a9c558cca3007ccea8754c9"
server_key_version: "encryptionkey@0"
I20260925 10:08:22.182235 18925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:22.192323 18925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:22.194882 18925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:22.195019 18925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:22.195310 18925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:22.195636 18925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:22.195672 18925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:22.195694 18925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:22.195706 18925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:22.201876 18925 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:41711
I20260925 10:08:22.201906 19052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:41711 every 8 connection(s)
I20260925 10:08:22.202250 18925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:08:22.204412 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 18925
I20260925 10:08:22.204496 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:08:22.204833 14501 external_mini_cluster.cc:1428] Setting key 2553fb93a4be0e0df828aafdf3ef6b48
Sep 25 10:08:22 dist-test-slave-s6tm krb5kdc[18553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330902, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:22.213574 18637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.203056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:56493 (local address 127.14.41.126:42259)
0925 10:08:22.203185 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:22.203187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:22.203423 (+ 236us) server_negotiation.cc:407] Connection header received
0925 10:08:22.204047 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:22.204048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:22.204061 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:22.204121 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:22.205420 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.206151 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.207310 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.207512 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.207583 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:22.210013 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:22.210015 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:22.210015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:22.210023 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:22.211735 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:22.212464 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:22.212466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:22.212466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:22.212533 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:22.212907 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:22.212909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:22.212909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:22.213044 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:22.213107 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:22.213383 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 10:08:22.213503 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:08:22.213769 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.203045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:42259 (local address 127.14.41.67:56493)
0925 10:08:22.203265 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:22.203304 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:22.203871 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:22.204225 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:22.204230 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:22.204590 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:22.205293 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:22.205302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.206256 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.206257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:22.207187 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:22.207190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.207605 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.207606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:22.207716 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:22.208344 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:22.208353 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:22.209860 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:22.211831 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:22.211838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:22.211840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:22.212361 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:22.212609 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:22.212610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:22.212610 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:22.212779 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:22.213110 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:22.213115 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:22.213245 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 10:08:22.213329 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 10:08:22.213519 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":75,"threads_started":1}
I20260925 10:08:22.214359 19053 heartbeater.cc:347] Connected to a master server at 127.14.41.126:42259
I20260925 10:08:22.214504 19053 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:22.214690 19053 heartbeater.cc:511] Master 127.14.41.126:42259 requested a full tablet report, sending...
I20260925 10:08:22.215145 18597 ts_manager.cc:194] Registered new tserver with Master: 2fb30134a8e0447fbe5c2e2bfa9ea856 (127.14.41.67:41711)
I20260925 10:08:22.215539 18597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:56493
I20260925 10:08:22.216871 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:08:22.224934 18637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.218219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.14.41.126:42259)
0925 10:08:22.218289 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:22.218291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:22.218551 (+ 260us) server_negotiation.cc:407] Connection header received
0925 10:08:22.218577 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:22.218578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:22.218588 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:22.218647 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:22.219236 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.219729 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.220546 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.220732 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.220791 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:22.222072 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:22.222074 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:22.222074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:22.222081 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:22.223481 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:22.223849 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:22.223851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:22.223851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:22.223914 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:22.224168 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:22.224169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:22.224169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:22.224353 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:22.224456 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:22.224622 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 10:08:22.224708 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:08:22.227260 18597 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:08:22.229161 18597 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:08:22.236965 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.232339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41430 (local address 127.14.41.67:41711)
0925 10:08:22.232541 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:22.232548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:22.232824 (+ 276us) server_negotiation.cc:407] Connection header received
0925 10:08:22.232958 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:22.232962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:22.232979 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:22.233038 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:22.233318 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.234162 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.235648 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.236405 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.236493 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:22.236562 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:22.236791 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 10:08:22.236841 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":93,"threads_started":1}
I20260925 10:08:22.237017 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.232635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58396 (local address 127.14.41.65:44707)
0925 10:08:22.232833 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:22.232838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:22.232883 (+ 45us) server_negotiation.cc:407] Connection header received
0925 10:08:22.233117 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:22.233120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:22.233136 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:22.233203 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:22.233401 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.234534 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.235856 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.236634 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.236740 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:22.236797 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:22.236889 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 10:08:22.236930 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":104,"threads_started":1}
I20260925 10:08:22.237236 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.232243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:41711 (local address 127.0.0.1:41430)
0925 10:08:22.232675 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:22.232738 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:22.232860 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:22.233168 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:22.233168 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:22.233171 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:22.233230 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:22.233232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.234285 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.234286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:22.235564 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:22.235567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.236513 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.236514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:22.236617 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:22.236633 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 10:08:22.237114 (+ 481us) client_negotiation.cc:240] Negotiation successful
0925 10:08:22.237163 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":4,"thread_start_us":69,"threads_started":1}
I20260925 10:08:22.237587 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.232528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:44707 (local address 127.0.0.1:58396)
0925 10:08:22.232830 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:22.232841 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:22.232893 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:22.233287 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:22.233288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:22.233291 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:22.233323 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:22.233325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.234697 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.234698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:22.235771 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:22.235773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.236744 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.236745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:22.236809 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:22.236814 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:08:22.236872 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 10:08:22.236999 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260925 10:08:22.238191 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.232492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35374 (local address 127.14.41.66:35623)
0925 10:08:22.232654 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:22.232658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:22.232921 (+ 263us) server_negotiation.cc:407] Connection header received
0925 10:08:22.232967 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:22.232970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:22.232986 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:22.233057 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:22.233359 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.234535 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.236952 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.237763 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.237836 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:22.237894 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:22.238095 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 10:08:22.238130 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260925 10:08:22.238183 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.232243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:35623 (local address 127.0.0.1:35374)
0925 10:08:22.232683 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:22.232741 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:22.232828 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:22.233132 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:22.233135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:22.233138 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:22.233220 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:22.233223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.235841 (+ 2618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.235842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:22.236860 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:22.236863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.237932 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.237933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:22.237997 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:22.238000 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:22.238060 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 10:08:22.238102 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":4,"spinlock_wait_cycles":768,"thread_start_us":92,"threads_started":1}
I20260925 10:08:22.238161 18987 tablet_service.cc:1467] Processing CreateTablet for tablet b01dcd3b2d8b455e982bb9ac1c4fffaa (DEFAULT_TABLE table=test-table [id=53843cd609914f8683f6a6059d05aad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:22.238507 18987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b01dcd3b2d8b455e982bb9ac1c4fffaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:22.240993 18854 tablet_service.cc:1467] Processing CreateTablet for tablet b01dcd3b2d8b455e982bb9ac1c4fffaa (DEFAULT_TABLE table=test-table [id=53843cd609914f8683f6a6059d05aad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:22.240993 18721 tablet_service.cc:1467] Processing CreateTablet for tablet b01dcd3b2d8b455e982bb9ac1c4fffaa (DEFAULT_TABLE table=test-table [id=53843cd609914f8683f6a6059d05aad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:22.241349 18854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b01dcd3b2d8b455e982bb9ac1c4fffaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:22.241350 18721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b01dcd3b2d8b455e982bb9ac1c4fffaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:22.241528 19070 tablet_bootstrap.cc:492] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856: Bootstrap starting.
I20260925 10:08:22.242126 19070 tablet_bootstrap.cc:654] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:22.242654 19070 log.cc:824] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:22.243413 19070 tablet_bootstrap.cc:492] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856: No bootstrap required, opened a new log
I20260925 10:08:22.243503 19070 ts_tablet_manager.cc:1397] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:08:22.244786 19072 tablet_bootstrap.cc:492] T b01dcd3b2d8b455e982bb9ac1c4fffaa P fb28bf62c00c4decb85310fa4a7a40ab: Bootstrap starting.
I20260925 10:08:22.245376 19070 raft_consensus.cc:348] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb30134a8e0447fbe5c2e2bfa9ea856" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 41711 } } peers { permanent_uuid: "fb28bf62c00c4decb85310fa4a7a40ab" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35623 } } peers { permanent_uuid: "4d670eecb9ca4447ae8d76637fe76841" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44707 } }
I20260925 10:08:22.245527 19072 tablet_bootstrap.cc:654] T b01dcd3b2d8b455e982bb9ac1c4fffaa P fb28bf62c00c4decb85310fa4a7a40ab: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:22.245574 19070 raft_consensus.cc:374] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:22.245613 19070 raft_consensus.cc:729] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb30134a8e0447fbe5c2e2bfa9ea856, State: Initialized, Role: FOLLOWER
I20260925 10:08:22.245759 19070 consensus_queue.cc:260] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb30134a8e0447fbe5c2e2bfa9ea856" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 41711 } } peers { permanent_uuid: "fb28bf62c00c4decb85310fa4a7a40ab" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35623 } } peers { permanent_uuid: "4d670eecb9ca4447ae8d76637fe76841" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44707 } }
I20260925 10:08:22.246161 19070 ts_tablet_manager.cc:1428] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:08:22.246186 19072 log.cc:824] T b01dcd3b2d8b455e982bb9ac1c4fffaa P fb28bf62c00c4decb85310fa4a7a40ab: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:22.246168 19053 heartbeater.cc:503] Master 127.14.41.126:42259 was elected leader, sending a full tablet report...
I20260925 10:08:22.246398 19073 tablet_bootstrap.cc:492] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 4d670eecb9ca4447ae8d76637fe76841: Bootstrap starting.
I20260925 10:08:22.247047 19072 tablet_bootstrap.cc:492] T b01dcd3b2d8b455e982bb9ac1c4fffaa P fb28bf62c00c4decb85310fa4a7a40ab: No bootstrap required, opened a new log
I20260925 10:08:22.247103 19073 tablet_bootstrap.cc:654] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 4d670eecb9ca4447ae8d76637fe76841: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:22.247186 19072 ts_tablet_manager.cc:1397] T b01dcd3b2d8b455e982bb9ac1c4fffaa P fb28bf62c00c4decb85310fa4a7a40ab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:08:22.247728 19073 log.cc:824] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 4d670eecb9ca4447ae8d76637fe76841: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:22.248545 19073 tablet_bootstrap.cc:492] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 4d670eecb9ca4447ae8d76637fe76841: No bootstrap required, opened a new log
I20260925 10:08:22.248618 19073 ts_tablet_manager.cc:1397] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 4d670eecb9ca4447ae8d76637fe76841: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:08:22.249480 19072 raft_consensus.cc:348] T b01dcd3b2d8b455e982bb9ac1c4fffaa P fb28bf62c00c4decb85310fa4a7a40ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb30134a8e0447fbe5c2e2bfa9ea856" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 41711 } } peers { permanent_uuid: "fb28bf62c00c4decb85310fa4a7a40ab" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35623 } } peers { permanent_uuid: "4d670eecb9ca4447ae8d76637fe76841" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44707 } }
I20260925 10:08:22.249652 19072 raft_consensus.cc:374] T b01dcd3b2d8b455e982bb9ac1c4fffaa P fb28bf62c00c4decb85310fa4a7a40ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:22.249691 19072 raft_consensus.cc:729] T b01dcd3b2d8b455e982bb9ac1c4fffaa P fb28bf62c00c4decb85310fa4a7a40ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb28bf62c00c4decb85310fa4a7a40ab, State: Initialized, Role: FOLLOWER
I20260925 10:08:22.249815 19072 consensus_queue.cc:260] T b01dcd3b2d8b455e982bb9ac1c4fffaa P fb28bf62c00c4decb85310fa4a7a40ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb30134a8e0447fbe5c2e2bfa9ea856" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 41711 } } peers { permanent_uuid: "fb28bf62c00c4decb85310fa4a7a40ab" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35623 } } peers { permanent_uuid: "4d670eecb9ca4447ae8d76637fe76841" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44707 } }
I20260925 10:08:22.250108 19072 ts_tablet_manager.cc:1428] T b01dcd3b2d8b455e982bb9ac1c4fffaa P fb28bf62c00c4decb85310fa4a7a40ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:08:22.250136 18920 heartbeater.cc:503] Master 127.14.41.126:42259 was elected leader, sending a full tablet report...
I20260925 10:08:22.250746 19073 raft_consensus.cc:348] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 4d670eecb9ca4447ae8d76637fe76841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb30134a8e0447fbe5c2e2bfa9ea856" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 41711 } } peers { permanent_uuid: "fb28bf62c00c4decb85310fa4a7a40ab" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35623 } } peers { permanent_uuid: "4d670eecb9ca4447ae8d76637fe76841" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44707 } }
I20260925 10:08:22.250895 19073 raft_consensus.cc:374] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 4d670eecb9ca4447ae8d76637fe76841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:22.250917 19073 raft_consensus.cc:729] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 4d670eecb9ca4447ae8d76637fe76841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d670eecb9ca4447ae8d76637fe76841, State: Initialized, Role: FOLLOWER
I20260925 10:08:22.251025 19073 consensus_queue.cc:260] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 4d670eecb9ca4447ae8d76637fe76841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb30134a8e0447fbe5c2e2bfa9ea856" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 41711 } } peers { permanent_uuid: "fb28bf62c00c4decb85310fa4a7a40ab" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35623 } } peers { permanent_uuid: "4d670eecb9ca4447ae8d76637fe76841" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44707 } }
I20260925 10:08:22.251319 19073 ts_tablet_manager.cc:1428] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 4d670eecb9ca4447ae8d76637fe76841: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 10:08:22.251335 18787 heartbeater.cc:503] Master 127.14.41.126:42259 was elected leader, sending a full tablet report...
W20260925 10:08:22.297571 18921 tablet.cc:2369] T b01dcd3b2d8b455e982bb9ac1c4fffaa P fb28bf62c00c4decb85310fa4a7a40ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:08:22.366312 18788 tablet.cc:2369] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 4d670eecb9ca4447ae8d76637fe76841: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:08:22.452862 19054 tablet.cc:2369] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:08:22.464488 19074 raft_consensus.cc:482] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:22.464682 19074 raft_consensus.cc:504] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb30134a8e0447fbe5c2e2bfa9ea856" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 41711 } } peers { permanent_uuid: "fb28bf62c00c4decb85310fa4a7a40ab" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35623 } } peers { permanent_uuid: "4d670eecb9ca4447ae8d76637fe76841" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44707 } }
I20260925 10:08:22.465207 19074 leader_election.cc:288] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb28bf62c00c4decb85310fa4a7a40ab (127.14.41.66:35623), 4d670eecb9ca4447ae8d76637fe76841 (127.14.41.65:44707)
I20260925 10:08:22.468283 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.465468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:35623 (local address 127.14.41.67:59789)
0925 10:08:22.465523 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:22.465537 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:22.465596 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:22.465783 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:22.465784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:22.465786 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:22.465856 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:22.465861 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.466567 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.466568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:22.467333 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:22.467335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.467999 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.468000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:22.468091 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:22.468094 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:22.468163 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 10:08:22.468207 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":1}
I20260925 10:08:22.468283 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.465584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:59789 (local address 127.14.41.66:35623)
0925 10:08:22.465659 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:22.465662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:22.465673 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:08:22.465703 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:22.465705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:22.465714 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:22.465772 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:22.465944 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.466473 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.467446 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.467839 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.467999 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:22.468010 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:22.468169 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 10:08:22.468207 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 10:08:22.468394 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.465632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:44707 (local address 127.14.41.67:48737)
0925 10:08:22.465794 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:22.465811 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:22.465889 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:22.466043 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:22.466044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:22.466045 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:22.466118 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:22.466119 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.466748 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.466749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:22.467506 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:22.467508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.468119 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.468120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:22.468198 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:22.468202 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:08:22.468271 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 10:08:22.468314 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":98,"threads_started":1}
I20260925 10:08:22.468394 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.465710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:48737 (local address 127.14.41.65:44707)
0925 10:08:22.465758 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:22.465760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:22.465881 (+ 121us) server_negotiation.cc:407] Connection header received
0925 10:08:22.465946 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:22.465947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:22.465956 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:22.466037 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:22.466198 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.466665 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.467603 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.468037 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.468120 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:22.468128 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:22.468291 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 10:08:22.468336 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:08:22.468704 18740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b01dcd3b2d8b455e982bb9ac1c4fffaa" candidate_uuid: "2fb30134a8e0447fbe5c2e2bfa9ea856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d670eecb9ca4447ae8d76637fe76841" is_pre_election: true
I20260925 10:08:22.468704 18874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b01dcd3b2d8b455e982bb9ac1c4fffaa" candidate_uuid: "2fb30134a8e0447fbe5c2e2bfa9ea856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb28bf62c00c4decb85310fa4a7a40ab" is_pre_election: true
I20260925 10:08:22.468861 18874 raft_consensus.cc:2454] T b01dcd3b2d8b455e982bb9ac1c4fffaa P fb28bf62c00c4decb85310fa4a7a40ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fb30134a8e0447fbe5c2e2bfa9ea856 in term 0.
I20260925 10:08:22.468879 18740 raft_consensus.cc:2454] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 4d670eecb9ca4447ae8d76637fe76841 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fb30134a8e0447fbe5c2e2bfa9ea856 in term 0.
I20260925 10:08:22.469112 18942 leader_election.cc:302] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb30134a8e0447fbe5c2e2bfa9ea856, fb28bf62c00c4decb85310fa4a7a40ab; no voters:
I20260925 10:08:22.469259 19074 raft_consensus.cc:2781] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:22.469336 19074 raft_consensus.cc:482] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:22.469360 19074 raft_consensus.cc:3037] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:22.470223 19074 raft_consensus.cc:504] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb30134a8e0447fbe5c2e2bfa9ea856" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 41711 } } peers { permanent_uuid: "fb28bf62c00c4decb85310fa4a7a40ab" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35623 } } peers { permanent_uuid: "4d670eecb9ca4447ae8d76637fe76841" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44707 } }
I20260925 10:08:22.470384 19074 leader_election.cc:288] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [CANDIDATE]: Term 1 election: Requested vote from peers fb28bf62c00c4decb85310fa4a7a40ab (127.14.41.66:35623), 4d670eecb9ca4447ae8d76637fe76841 (127.14.41.65:44707)
I20260925 10:08:22.470594 18874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b01dcd3b2d8b455e982bb9ac1c4fffaa" candidate_uuid: "2fb30134a8e0447fbe5c2e2bfa9ea856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb28bf62c00c4decb85310fa4a7a40ab"
I20260925 10:08:22.470667 18740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b01dcd3b2d8b455e982bb9ac1c4fffaa" candidate_uuid: "2fb30134a8e0447fbe5c2e2bfa9ea856" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d670eecb9ca4447ae8d76637fe76841"
I20260925 10:08:22.470695 18874 raft_consensus.cc:3037] T b01dcd3b2d8b455e982bb9ac1c4fffaa P fb28bf62c00c4decb85310fa4a7a40ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:22.470779 18740 raft_consensus.cc:3037] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 4d670eecb9ca4447ae8d76637fe76841 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:22.471499 18874 raft_consensus.cc:2454] T b01dcd3b2d8b455e982bb9ac1c4fffaa P fb28bf62c00c4decb85310fa4a7a40ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fb30134a8e0447fbe5c2e2bfa9ea856 in term 1.
I20260925 10:08:22.471525 18740 raft_consensus.cc:2454] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 4d670eecb9ca4447ae8d76637fe76841 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fb30134a8e0447fbe5c2e2bfa9ea856 in term 1.
I20260925 10:08:22.471701 18942 leader_election.cc:302] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb30134a8e0447fbe5c2e2bfa9ea856, 4d670eecb9ca4447ae8d76637fe76841; no voters:
I20260925 10:08:22.471817 19074 raft_consensus.cc:2781] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:22.472033 19074 raft_consensus.cc:686] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [term 1 LEADER]: Becoming Leader. State: Replica: 2fb30134a8e0447fbe5c2e2bfa9ea856, State: Running, Role: LEADER
I20260925 10:08:22.472165 19074 consensus_queue.cc:237] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb30134a8e0447fbe5c2e2bfa9ea856" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 41711 } } peers { permanent_uuid: "fb28bf62c00c4decb85310fa4a7a40ab" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35623 } } peers { permanent_uuid: "4d670eecb9ca4447ae8d76637fe76841" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44707 } }
I20260925 10:08:22.473064 18597 catalog_manager.cc:5495] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fb30134a8e0447fbe5c2e2bfa9ea856 (127.14.41.67). New cstate: current_term: 1 leader_uuid: "2fb30134a8e0447fbe5c2e2bfa9ea856" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb30134a8e0447fbe5c2e2bfa9ea856" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 41711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb28bf62c00c4decb85310fa4a7a40ab" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d670eecb9ca4447ae8d76637fe76841" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44707 } health_report { overall_health: UNKNOWN } } }
I20260925 10:08:22.526077 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.523292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41432 (local address 127.14.41.67:41711)
0925 10:08:22.523376 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:22.523378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:22.523389 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:08:22.523438 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:22.523440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:22.523458 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:22.523524 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:22.523676 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.524245 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.525133 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:22.525345 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:22.525429 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:22.525587 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:22.525754 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:22.525823 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:22.525910 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 10:08:22.525946 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:08:22.528045 18740 raft_consensus.cc:1261] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 4d670eecb9ca4447ae8d76637fe76841 [term 1 FOLLOWER]: Refusing update from remote peer 2fb30134a8e0447fbe5c2e2bfa9ea856: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:22.528106 18874 raft_consensus.cc:1261] T b01dcd3b2d8b455e982bb9ac1c4fffaa P fb28bf62c00c4decb85310fa4a7a40ab [term 1 FOLLOWER]: Refusing update from remote peer 2fb30134a8e0447fbe5c2e2bfa9ea856: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:22.528416 19074 consensus_queue.cc:1035] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d670eecb9ca4447ae8d76637fe76841" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:22.528571 19080 consensus_queue.cc:1035] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb28bf62c00c4decb85310fa4a7a40ab" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:22.531481 19083 mvcc.cc:205] Tried to move back new op lower bound from 7333195376751771648 to 7333195376526594048. Current Snapshot: MvccSnapshot[applied={T|T < 7333195376751771648 or (T in {7333195376751771648})}]
I20260925 10:08:22.531607 19085 mvcc.cc:205] Tried to move back new op lower bound from 7333195376751771648 to 7333195376526594048. Current Snapshot: MvccSnapshot[applied={T|T < 7333195376751771648}]
I20260925 10:08:22.533486 18596 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:22.533587 18596 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:22.534494 18596 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62036e9eae7a4f93b1eb263a22e5f278: Sending DeleteTablet for 3 replicas of tablet b01dcd3b2d8b455e982bb9ac1c4fffaa
I20260925 10:08:22.534821 18987 tablet_service.cc:1514] Processing DeleteTablet for tablet b01dcd3b2d8b455e982bb9ac1c4fffaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:22 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:41430
I20260925 10:08:22.534933 18721 tablet_service.cc:1514] Processing DeleteTablet for tablet b01dcd3b2d8b455e982bb9ac1c4fffaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:22 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:58396
I20260925 10:08:22.534884 18854 tablet_service.cc:1514] Processing DeleteTablet for tablet b01dcd3b2d8b455e982bb9ac1c4fffaa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:22 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:35374
I20260925 10:08:22.535166 19092 tablet_replica.cc:330] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856: stopping tablet replica
I20260925 10:08:22.535300 19092 raft_consensus.cc:2229] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:08:22.535403 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 18659
I20260925 10:08:22.535463 19092 raft_consensus.cc:2258] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:22.537942 19092 ts_tablet_manager.cc:1905] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:22.539710 19092 ts_tablet_manager.cc:1918] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:08:22.539839 19092 log.cc:1196] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/b01dcd3b2d8b455e982bb9ac1c4fffaa
I20260925 10:08:22.540208 19092 ts_tablet_manager.cc:1939] T b01dcd3b2d8b455e982bb9ac1c4fffaa P 2fb30134a8e0447fbe5c2e2bfa9ea856: Deleting consensus metadata
W20260925 10:08:22.540779 18583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.41.65:44707 (error 108)
I20260925 10:08:22.541033 19093 tablet_replica.cc:330] T b01dcd3b2d8b455e982bb9ac1c4fffaa P fb28bf62c00c4decb85310fa4a7a40ab: stopping tablet replica
I20260925 10:08:22.541174 19093 raft_consensus.cc:2229] T b01dcd3b2d8b455e982bb9ac1c4fffaa P fb28bf62c00c4decb85310fa4a7a40ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:22.541188 18582 catalog_manager.cc:4841] TS 2fb30134a8e0447fbe5c2e2bfa9ea856 (127.14.41.67:41711): tablet b01dcd3b2d8b455e982bb9ac1c4fffaa (table test-table [id=53843cd609914f8683f6a6059d05aad1]) successfully deleted
I20260925 10:08:22.541338 19093 raft_consensus.cc:2258] T b01dcd3b2d8b455e982bb9ac1c4fffaa P fb28bf62c00c4decb85310fa4a7a40ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:22.541689 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.541554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:44707 (local address 127.0.0.1:58408)
0925 10:08:22.541603 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:22.541650 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.65:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 10:08:22.541675 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 18792
W20260925 10:08:22.544617 18583 catalog_manager.cc:4584] TS 4d670eecb9ca4447ae8d76637fe76841 (127.14.41.65:44707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b01dcd3b2d8b455e982bb9ac1c4fffaa: Network error: Client connection negotiation failed: client connection to 127.14.41.65:44707: connect: Connection refused (error 111)
I20260925 10:08:22.548280 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.548120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:35623 (local address 127.0.0.1:35384)
0925 10:08:22.548195 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:22.548234 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.66:35623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 10:08:22.548280 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 18925
I20260925 10:08:22.553299 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 18566
2026-09-25T10:08:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2277 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T10:08:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:22Z Disabled control of system clock
I20260925 10:08:22.582537 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:37589
--webserver_interface=127.14.41.126
--webserver_port=0
--builtin_ntp_servers=127.14.41.84:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:37589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 10:08:22.669523 19099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:22.669754 19099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:22.669775 19099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:22.670887 19099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:22.670921 19099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:22.670934 19099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:22.670948 19099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:22.670959 19099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:08:22.672345 19099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:37589
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:37589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:22.672557 19099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:22.672886 19099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:22.678071 19105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:22.678082 19104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:22.678082 19107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:22.678166 19099 server_base.cc:1034] running on GCE node
I20260925 10:08:22.678539 19099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:22.678789 19099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:22.679987 19099 hybrid_clock.cc:648] HybridClock initialized: now 1790330902679951 us; error 45 us; skew 500 ppm
I20260925 10:08:22.681247 19099 webserver.cc:460] Webserver started at http://127.14.41.126:44021/ using document root <none> and password file <none>
I20260925 10:08:22.681531 19099 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:22.681607 19099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:22.681761 19099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:22.682876 19099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "c829a82082384557a9029323091c0dd1"
format_stamp: "Formatted at 2026-09-25 10:08:22 on dist-test-slave-s6tm"
server_key: "9591379fa4ce390a735c4f43a097afe7"
server_key_iv: "ab21107ceebe092c0cb1e7209b2ebcbe"
server_key_version: "encryptionkey@0"
I20260925 10:08:22.683249 19099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "c829a82082384557a9029323091c0dd1"
format_stamp: "Formatted at 2026-09-25 10:08:22 on dist-test-slave-s6tm"
server_key: "9591379fa4ce390a735c4f43a097afe7"
server_key_iv: "ab21107ceebe092c0cb1e7209b2ebcbe"
server_key_version: "encryptionkey@0"
I20260925 10:08:22.684787 19099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:08:22.685665 19113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:22.685935 19099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:22.686046 19099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "c829a82082384557a9029323091c0dd1"
format_stamp: "Formatted at 2026-09-25 10:08:22 on dist-test-slave-s6tm"
server_key: "9591379fa4ce390a735c4f43a097afe7"
server_key_iv: "ab21107ceebe092c0cb1e7209b2ebcbe"
server_key_version: "encryptionkey@0"
I20260925 10:08:22.686125 19099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:22.689932 19099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:22.690240 19099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:22.690397 19099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:22.694535 19099 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:37589
I20260925 10:08:22.694571 19165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:37589 every 8 connection(s)
I20260925 10:08:22.694960 19099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:22.695636 19166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:22.697342 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 19099
I20260925 10:08:22.697508 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:08:22.697685 14501 external_mini_cluster.cc:1428] Setting key bfbb1db58ee41320597665698abd85cd
I20260925 10:08:22.699126 19166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1: Bootstrap starting.
I20260925 10:08:22.699851 19166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:22.700197 19166 log.cc:824] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:22.700572 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.698277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57608 (local address 127.14.41.126:37589)
0925 10:08:22.698551 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:22.698560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:22.698597 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:08:22.699111 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:22.699122 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:22.699176 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:22.699329 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:22.700097 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:22.700100 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:22.700101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:22.700105 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:22.700158 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:22.700168 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:22.700239 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:22.700309 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 10:08:22.700376 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":138,"threads_started":1}
I20260925 10:08:22.701011 19166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1: No bootstrap required, opened a new log
I20260925 10:08:22.702508 19166 raft_consensus.cc:348] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c829a82082384557a9029323091c0dd1" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37589 } }
I20260925 10:08:22.702692 19166 raft_consensus.cc:374] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:22.702735 19166 raft_consensus.cc:729] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c829a82082384557a9029323091c0dd1, State: Initialized, Role: FOLLOWER
I20260925 10:08:22.702848 19166 consensus_queue.cc:260] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c829a82082384557a9029323091c0dd1" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37589 } }
I20260925 10:08:22.702926 19166 raft_consensus.cc:388] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:08:22.702948 19166 raft_consensus.cc:482] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:22.702982 19166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:22.703760 19166 raft_consensus.cc:504] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c829a82082384557a9029323091c0dd1" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37589 } }
I20260925 10:08:22.703907 19166 leader_election.cc:302] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c829a82082384557a9029323091c0dd1; no voters:
I20260925 10:08:22.704099 19166 leader_election.cc:288] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:08:22.704183 19171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:22.704396 19171 raft_consensus.cc:686] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1 [term 1 LEADER]: Becoming Leader. State: Replica: c829a82082384557a9029323091c0dd1, State: Running, Role: LEADER
I20260925 10:08:22.704437 19166 sys_catalog.cc:564] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:22.704530 19171 consensus_queue.cc:237] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c829a82082384557a9029323091c0dd1" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37589 } }
I20260925 10:08:22.704906 19172 sys_catalog.cc:455] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c829a82082384557a9029323091c0dd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c829a82082384557a9029323091c0dd1" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37589 } } }
I20260925 10:08:22.705078 19172 sys_catalog.cc:458] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:22.704911 19173 sys_catalog.cc:455] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c829a82082384557a9029323091c0dd1. Latest consensus state: current_term: 1 leader_uuid: "c829a82082384557a9029323091c0dd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c829a82082384557a9029323091c0dd1" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37589 } } }
I20260925 10:08:22.705307 19173 sys_catalog.cc:458] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:22.705471 19179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:22.706169 19179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:22.708047 19179 catalog_manager.cc:1347] Generated new cluster ID: 46089ee0331f41339021e138b426210d
I20260925 10:08:22.708108 19179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:08:22.722137 19179 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:08:22.722359 19179 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:22.725461 19179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1: Generated new TSK 0
I20260925 10:08:22.725675 19179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:08:22.731742 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37589
--builtin_ntp_servers=127.14.41.84:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 10:08:22.825006 19190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:22.825269 19190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:22.825313 19190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:22.826629 19190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:22.826711 19190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:22.826759 19190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:08:22.828411 19190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:22.828698 19190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:22.829030 19190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:22.835490 19198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:22.835603 19196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:22.835623 19195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:22.835752 19190 server_base.cc:1034] running on GCE node
I20260925 10:08:22.836354 19190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:22.836719 19190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:22.837939 19190 hybrid_clock.cc:648] HybridClock initialized: now 1790330902837924 us; error 30 us; skew 500 ppm
I20260925 10:08:22.839488 19190 webserver.cc:460] Webserver started at http://127.14.41.65:41445/ using document root <none> and password file <none>
I20260925 10:08:22.839867 19190 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:22.839977 19190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:22.840137 19190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:22.841359 19190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "b9c52203f34d4be5b9bf9591f5b00735"
format_stamp: "Formatted at 2026-09-25 10:08:22 on dist-test-slave-s6tm"
server_key: "b0ed4fcccd8cb6315b0d615477eb1700"
server_key_iv: "c8fa3701915e5fa559fbe8c930a22ace"
server_key_version: "encryptionkey@0"
I20260925 10:08:22.841789 19190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "b9c52203f34d4be5b9bf9591f5b00735"
format_stamp: "Formatted at 2026-09-25 10:08:22 on dist-test-slave-s6tm"
server_key: "b0ed4fcccd8cb6315b0d615477eb1700"
server_key_iv: "c8fa3701915e5fa559fbe8c930a22ace"
server_key_version: "encryptionkey@0"
I20260925 10:08:22.843241 19190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:08:22.844105 19204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:22.844267 19190 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:22.844374 19190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "b9c52203f34d4be5b9bf9591f5b00735"
format_stamp: "Formatted at 2026-09-25 10:08:22 on dist-test-slave-s6tm"
server_key: "b0ed4fcccd8cb6315b0d615477eb1700"
server_key_iv: "c8fa3701915e5fa559fbe8c930a22ace"
server_key_version: "encryptionkey@0"
I20260925 10:08:22.844456 19190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:22.848073 19190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:22.848374 19190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:22.848512 19190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:22.848762 19190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:22.849135 19190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:22.849170 19190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:22.849192 19190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:22.849246 19190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:22.856171 19190 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:42413
I20260925 10:08:22.856218 19317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:42413 every 8 connection(s)
I20260925 10:08:22.856580 19190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:08:22.857554 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 19190
I20260925 10:08:22.857656 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:08:22.857872 14501 external_mini_cluster.cc:1428] Setting key 9ac765e6e7a69c1b71274b7e5dc13d2a
I20260925 10:08:22.858949 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37589
--builtin_ntp_servers=127.14.41.84:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 10:08:22.860297 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.857342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:52589 (local address 127.14.41.126:37589)
0925 10:08:22.857462 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:22.857465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:22.857753 (+ 288us) server_negotiation.cc:407] Connection header received
0925 10:08:22.858399 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:22.858400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:22.858406 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:22.858465 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:22.859910 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:22.859911 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:22.859912 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:22.859917 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:22.860043 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:22.860049 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:22.860101 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:22.860211 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 10:08:22.860227 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 10:08:22.860535 19320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.857307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:37589 (local address 127.14.41.65:52589)
0925 10:08:22.857580 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:22.857636 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:22.858275 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:22.858545 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:22.858549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:22.858565 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:22.859726 (+ 1161us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:08:22.859727 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:22.859758 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:08:22.859791 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:22.860113 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:22.860120 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:22.860124 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:08:22.860223 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 10:08:22.860291 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":137,"threads_started":1}
I20260925 10:08:22.861160 19318 heartbeater.cc:347] Connected to a master server at 127.14.41.126:37589
I20260925 10:08:22.861272 19318 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:22.861439 19318 heartbeater.cc:511] Master 127.14.41.126:37589 requested a full tablet report, sending...
I20260925 10:08:22.862010 19130 ts_manager.cc:194] Registered new tserver with Master: b9c52203f34d4be5b9bf9591f5b00735 (127.14.41.65:42413)
W20260925 10:08:22.945060 19321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:22.945281 19321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:22.945307 19321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:22.946718 19321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:22.946776 19321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:22.946815 19321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:08:22.948928 19321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:22.949239 19321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:22.949587 19321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:22.954957 19326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:22.955020 19327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:22.954994 19329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:22.955312 19321 server_base.cc:1034] running on GCE node
I20260925 10:08:22.955528 19321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:22.955786 19321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:22.956996 19321 hybrid_clock.cc:648] HybridClock initialized: now 1790330902956976 us; error 48 us; skew 500 ppm
I20260925 10:08:22.958210 19321 webserver.cc:460] Webserver started at http://127.14.41.66:43433/ using document root <none> and password file <none>
I20260925 10:08:22.958477 19321 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:22.958562 19321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:22.958720 19321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:22.959900 19321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "08bab9891e984d4a823706b0661c440d"
format_stamp: "Formatted at 2026-09-25 10:08:22 on dist-test-slave-s6tm"
server_key: "76aca2ae7688d577280c6c26a03ffda1"
server_key_iv: "fe7f761dccf96d47eb67dca5af60048d"
server_key_version: "encryptionkey@0"
I20260925 10:08:22.960336 19321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "08bab9891e984d4a823706b0661c440d"
format_stamp: "Formatted at 2026-09-25 10:08:22 on dist-test-slave-s6tm"
server_key: "76aca2ae7688d577280c6c26a03ffda1"
server_key_iv: "fe7f761dccf96d47eb67dca5af60048d"
server_key_version: "encryptionkey@0"
I20260925 10:08:22.961794 19321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:08:22.962639 19335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:22.962829 19321 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:22.963022 19321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "08bab9891e984d4a823706b0661c440d"
format_stamp: "Formatted at 2026-09-25 10:08:22 on dist-test-slave-s6tm"
server_key: "76aca2ae7688d577280c6c26a03ffda1"
server_key_iv: "fe7f761dccf96d47eb67dca5af60048d"
server_key_version: "encryptionkey@0"
I20260925 10:08:22.963106 19321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:22.966728 19321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:22.967031 19321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:22.967182 19321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:22.967413 19321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:22.967747 19321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:22.967811 19321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:22.967860 19321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:22.967887 19321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:22.974231 19321 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:45915
I20260925 10:08:22.974341 19448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:45915 every 8 connection(s)
I20260925 10:08:22.974689 19321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:08:22.978004 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.975506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:45827 (local address 127.14.41.126:37589)
0925 10:08:22.975572 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:22.975574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:22.975779 (+ 205us) server_negotiation.cc:407] Connection header received
0925 10:08:22.976430 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:22.976431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:22.976437 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:22.976491 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:22.977742 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:22.977744 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:22.977744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:22.977748 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:22.977763 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:22.977767 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:22.977820 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:22.977933 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 10:08:22.977946 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 10:08:22.978166 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:22.975507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:37589 (local address 127.14.41.66:45827)
0925 10:08:22.975673 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:22.975707 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:22.976321 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:22.976550 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:22.976554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:22.976563 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:22.977584 (+ 1021us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:08:22.977584 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:22.977614 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:08:22.977645 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:22.977833 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:22.977837 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:22.977839 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:08:22.977899 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 10:08:22.977961 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260925 10:08:22.978713 19449 heartbeater.cc:347] Connected to a master server at 127.14.41.126:37589
I20260925 10:08:22.978802 19449 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:22.978916 19449 heartbeater.cc:511] Master 127.14.41.126:37589 requested a full tablet report, sending...
I20260925 10:08:22.979323 19130 ts_manager.cc:194] Registered new tserver with Master: 08bab9891e984d4a823706b0661c440d (127.14.41.66:45915)
I20260925 10:08:22.984570 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 19321
I20260925 10:08:22.984695 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:08:22.984903 14501 external_mini_cluster.cc:1428] Setting key 5c8688845ca2ff5d0226460c8a15d78b
I20260925 10:08:22.986354 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37589
--builtin_ntp_servers=127.14.41.84:42627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 10:08:23.071513 19452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:23.071753 19452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:23.071779 19452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:23.073107 19452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:23.073158 19452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:23.073186 19452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:08:23.074667 19452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:42627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:23.074949 19452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:23.075286 19452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:23.080755 19457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:23.080789 19452 server_base.cc:1034] running on GCE node
W20260925 10:08:23.080790 19458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:23.080755 19460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:23.081331 19452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:23.081560 19452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:23.082743 19452 hybrid_clock.cc:648] HybridClock initialized: now 1790330903082718 us; error 53 us; skew 500 ppm
I20260925 10:08:23.084170 19452 webserver.cc:460] Webserver started at http://127.14.41.67:42141/ using document root <none> and password file <none>
I20260925 10:08:23.084511 19452 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:23.084590 19452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:23.084748 19452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:23.085862 19452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "4cf96a45477f4cb8b065ed6b1ba8914c"
format_stamp: "Formatted at 2026-09-25 10:08:23 on dist-test-slave-s6tm"
server_key: "0c0d3202d99a37f20c2f7643737bfb62"
server_key_iv: "ad1ecad452f806a6d3958b6c8ef11c68"
server_key_version: "encryptionkey@0"
I20260925 10:08:23.086305 19452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "4cf96a45477f4cb8b065ed6b1ba8914c"
format_stamp: "Formatted at 2026-09-25 10:08:23 on dist-test-slave-s6tm"
server_key: "0c0d3202d99a37f20c2f7643737bfb62"
server_key_iv: "ad1ecad452f806a6d3958b6c8ef11c68"
server_key_version: "encryptionkey@0"
I20260925 10:08:23.087733 19452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:08:23.088675 19466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:23.088876 19452 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:23.089000 19452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "4cf96a45477f4cb8b065ed6b1ba8914c"
format_stamp: "Formatted at 2026-09-25 10:08:23 on dist-test-slave-s6tm"
server_key: "0c0d3202d99a37f20c2f7643737bfb62"
server_key_iv: "ad1ecad452f806a6d3958b6c8ef11c68"
server_key_version: "encryptionkey@0"
I20260925 10:08:23.089095 19452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:23.092994 19452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:23.093309 19452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:23.093466 19452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:23.093681 19452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:23.094002 19452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:23.094058 19452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:23.094099 19452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:23.094126 19452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:23.100056 19452 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:33867
I20260925 10:08:23.100096 19579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:33867 every 8 connection(s)
I20260925 10:08:23.100474 19452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:08:23.101522 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 19452
I20260925 10:08:23.101625 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:08:23.101827 14501 external_mini_cluster.cc:1428] Setting key 26271828f3b01dd826055c695951d148
I20260925 10:08:23.103895 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.101343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:41309 (local address 127.14.41.126:37589)
0925 10:08:23.101409 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:23.101413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:23.101653 (+ 240us) server_negotiation.cc:407] Connection header received
0925 10:08:23.102371 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:23.102372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:23.102378 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:23.102424 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:23.103686 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:23.103687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:23.103687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:23.103692 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:23.103709 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:23.103713 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:23.103762 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:23.103831 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 10:08:23.103843 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 10:08:23.104235 19582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.101291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:37589 (local address 127.14.41.67:41309)
0925 10:08:23.101523 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:23.101561 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:23.102252 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:23.102499 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:23.102504 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:23.102515 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:23.103536 (+ 1021us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:08:23.103536 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:23.103567 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:08:23.103596 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:23.103774 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:23.103778 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:23.103781 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:23.103839 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 10:08:23.104010 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260925 10:08:23.104790 19580 heartbeater.cc:347] Connected to a master server at 127.14.41.126:37589
I20260925 10:08:23.104907 19580 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:23.105060 19580 heartbeater.cc:511] Master 127.14.41.126:37589 requested a full tablet report, sending...
I20260925 10:08:23.105528 19130 ts_manager.cc:194] Registered new tserver with Master: 4cf96a45477f4cb8b065ed6b1ba8914c (127.14.41.67:33867)
I20260925 10:08:23.114023 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:08:23.116729 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.115457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57622 (local address 127.14.41.126:37589)
0925 10:08:23.115530 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:23.115533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:23.115625 (+ 92us) server_negotiation.cc:407] Connection header received
0925 10:08:23.115671 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:23.115672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:23.115677 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:23.115725 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:23.116463 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:23.116465 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:23.116465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:23.116469 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:23.116484 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:23.116488 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:23.116550 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:23.116586 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 10:08:23.116597 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:08:23.118549 19130 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:08:23.120296 19130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:08:23.124874 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.123020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55358 (local address 127.14.41.66:45915)
0925 10:08:23.123203 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:23.123206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:23.123371 (+ 165us) server_negotiation.cc:407] Connection header received
0925 10:08:23.123403 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:23.123406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:23.123416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:23.123482 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:23.124493 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:23.124501 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:23.124502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:23.124506 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:23.124542 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:23.124573 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:23.124638 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:23.124741 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 10:08:23.124757 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
I20260925 10:08:23.124887 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.122837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41962 (local address 127.14.41.67:33867)
0925 10:08:23.123033 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:23.123039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:23.123129 (+ 90us) server_negotiation.cc:407] Connection header received
0925 10:08:23.123204 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:23.123211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:23.123222 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:23.123435 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:23.124506 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:23.124514 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:23.124515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:23.124521 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:23.124557 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:23.124584 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:23.124679 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:23.124781 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 10:08:23.124797 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
I20260925 10:08:23.124825 19590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.122651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:45915 (local address 127.0.0.1:55358)
0925 10:08:23.123258 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:23.123267 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:23.123305 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:23.123799 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:23.123800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:23.123808 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:23.124397 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:08:23.124397 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:23.124402 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:08:23.124411 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:23.124649 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:23.124656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:23.124659 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:23.124732 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 10:08:23.124748 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"spinlock_wait_cycles":384,"thread_start_us":121,"threads_started":1}
I20260925 10:08:23.125063 19588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.122647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:33867 (local address 127.0.0.1:41962)
0925 10:08:23.123021 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:23.123065 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:23.123142 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:23.123799 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:23.123800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:23.123805 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:23.124336 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:08:23.124336 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:23.124367 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:08:23.124394 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:23.124703 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:23.124704 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:23.124704 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 10:08:23.124980 (+ 276us) client_negotiation.cc:240] Negotiation successful
0925 10:08:23.124994 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"spinlock_wait_cycles":768,"thread_start_us":108,"threads_started":1}
I20260925 10:08:23.125106 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.122911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35300 (local address 127.14.41.65:42413)
0925 10:08:23.123120 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:23.123125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:23.123324 (+ 199us) server_negotiation.cc:407] Connection header received
0925 10:08:23.123353 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:23.123356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:23.123366 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:23.123448 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:23.124636 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:23.124643 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:23.124643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:23.124648 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:23.124680 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:23.124703 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:23.124757 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:23.124962 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 10:08:23.124982 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":98,"threads_started":1}
I20260925 10:08:23.125264 19589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.122647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:42413 (local address 127.0.0.1:35300)
0925 10:08:23.123221 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:23.123232 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:23.123288 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:23.123792 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:23.123797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:23.123803 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:23.124542 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:08:23.124543 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:23.124547 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:08:23.124555 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:23.124782 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:23.124782 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:23.124783 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 10:08:23.124825 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 10:08:23.124837 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260925 10:08:23.126019 19252 tablet_service.cc:1467] Processing CreateTablet for tablet bcc18273ac5741369d7f109e823f52f5 (DEFAULT_TABLE table=test-table [id=3e16e0d9170649cfb9aa42807f288049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:23.126032 19514 tablet_service.cc:1467] Processing CreateTablet for tablet bcc18273ac5741369d7f109e823f52f5 (DEFAULT_TABLE table=test-table [id=3e16e0d9170649cfb9aa42807f288049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:23.126206 19383 tablet_service.cc:1467] Processing CreateTablet for tablet bcc18273ac5741369d7f109e823f52f5 (DEFAULT_TABLE table=test-table [id=3e16e0d9170649cfb9aa42807f288049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:23.126394 19514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcc18273ac5741369d7f109e823f52f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:23.126401 19252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcc18273ac5741369d7f109e823f52f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:23.126513 19383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcc18273ac5741369d7f109e823f52f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:23.129042 19594 tablet_bootstrap.cc:492] T bcc18273ac5741369d7f109e823f52f5 P b9c52203f34d4be5b9bf9591f5b00735: Bootstrap starting.
I20260925 10:08:23.129698 19595 tablet_bootstrap.cc:492] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d: Bootstrap starting.
I20260925 10:08:23.129873 19594 tablet_bootstrap.cc:654] T bcc18273ac5741369d7f109e823f52f5 P b9c52203f34d4be5b9bf9591f5b00735: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:23.130187 19594 log.cc:824] T bcc18273ac5741369d7f109e823f52f5 P b9c52203f34d4be5b9bf9591f5b00735: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:23.130316 19595 tablet_bootstrap.cc:654] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:23.130657 19595 log.cc:824] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:23.130981 19596 tablet_bootstrap.cc:492] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c: Bootstrap starting.
I20260925 10:08:23.131034 19594 tablet_bootstrap.cc:492] T bcc18273ac5741369d7f109e823f52f5 P b9c52203f34d4be5b9bf9591f5b00735: No bootstrap required, opened a new log
I20260925 10:08:23.131119 19594 ts_tablet_manager.cc:1397] T bcc18273ac5741369d7f109e823f52f5 P b9c52203f34d4be5b9bf9591f5b00735: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:08:23.131372 19595 tablet_bootstrap.cc:492] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d: No bootstrap required, opened a new log
I20260925 10:08:23.131441 19595 ts_tablet_manager.cc:1397] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:08:23.131666 19596 tablet_bootstrap.cc:654] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:23.132007 19596 log.cc:824] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:23.132761 19596 tablet_bootstrap.cc:492] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c: No bootstrap required, opened a new log
I20260925 10:08:23.132826 19596 ts_tablet_manager.cc:1397] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:08:23.133651 19595 raft_consensus.cc:348] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf96a45477f4cb8b065ed6b1ba8914c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33867 } } peers { permanent_uuid: "08bab9891e984d4a823706b0661c440d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45915 } } peers { permanent_uuid: "b9c52203f34d4be5b9bf9591f5b00735" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42413 } }
I20260925 10:08:23.133770 19595 raft_consensus.cc:374] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:23.133790 19595 raft_consensus.cc:729] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08bab9891e984d4a823706b0661c440d, State: Initialized, Role: FOLLOWER
I20260925 10:08:23.133824 19594 raft_consensus.cc:348] T bcc18273ac5741369d7f109e823f52f5 P b9c52203f34d4be5b9bf9591f5b00735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf96a45477f4cb8b065ed6b1ba8914c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33867 } } peers { permanent_uuid: "08bab9891e984d4a823706b0661c440d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45915 } } peers { permanent_uuid: "b9c52203f34d4be5b9bf9591f5b00735" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42413 } }
I20260925 10:08:23.133922 19595 consensus_queue.cc:260] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf96a45477f4cb8b065ed6b1ba8914c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33867 } } peers { permanent_uuid: "08bab9891e984d4a823706b0661c440d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45915 } } peers { permanent_uuid: "b9c52203f34d4be5b9bf9591f5b00735" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42413 } }
I20260925 10:08:23.133993 19594 raft_consensus.cc:374] T bcc18273ac5741369d7f109e823f52f5 P b9c52203f34d4be5b9bf9591f5b00735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:23.134030 19594 raft_consensus.cc:729] T bcc18273ac5741369d7f109e823f52f5 P b9c52203f34d4be5b9bf9591f5b00735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9c52203f34d4be5b9bf9591f5b00735, State: Initialized, Role: FOLLOWER
I20260925 10:08:23.134157 19594 consensus_queue.cc:260] T bcc18273ac5741369d7f109e823f52f5 P b9c52203f34d4be5b9bf9591f5b00735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf96a45477f4cb8b065ed6b1ba8914c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33867 } } peers { permanent_uuid: "08bab9891e984d4a823706b0661c440d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45915 } } peers { permanent_uuid: "b9c52203f34d4be5b9bf9591f5b00735" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42413 } }
I20260925 10:08:23.134203 19595 ts_tablet_manager.cc:1428] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:08:23.134438 19594 ts_tablet_manager.cc:1428] T bcc18273ac5741369d7f109e823f52f5 P b9c52203f34d4be5b9bf9591f5b00735: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 10:08:23.134464 19449 heartbeater.cc:503] Master 127.14.41.126:37589 was elected leader, sending a full tablet report...
I20260925 10:08:23.134581 19318 heartbeater.cc:503] Master 127.14.41.126:37589 was elected leader, sending a full tablet report...
I20260925 10:08:23.135716 19596 raft_consensus.cc:348] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf96a45477f4cb8b065ed6b1ba8914c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33867 } } peers { permanent_uuid: "08bab9891e984d4a823706b0661c440d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45915 } } peers { permanent_uuid: "b9c52203f34d4be5b9bf9591f5b00735" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42413 } }
I20260925 10:08:23.135895 19596 raft_consensus.cc:374] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:23.135972 19596 raft_consensus.cc:729] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf96a45477f4cb8b065ed6b1ba8914c, State: Initialized, Role: FOLLOWER
I20260925 10:08:23.136091 19596 consensus_queue.cc:260] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf96a45477f4cb8b065ed6b1ba8914c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33867 } } peers { permanent_uuid: "08bab9891e984d4a823706b0661c440d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45915 } } peers { permanent_uuid: "b9c52203f34d4be5b9bf9591f5b00735" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42413 } }
I20260925 10:08:23.136382 19596 ts_tablet_manager.cc:1428] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260925 10:08:23.136435 19580 heartbeater.cc:503] Master 127.14.41.126:37589 was elected leader, sending a full tablet report...
I20260925 10:08:23.213572 19600 raft_consensus.cc:482] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:23.213871 19600 raft_consensus.cc:504] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf96a45477f4cb8b065ed6b1ba8914c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33867 } } peers { permanent_uuid: "08bab9891e984d4a823706b0661c440d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45915 } } peers { permanent_uuid: "b9c52203f34d4be5b9bf9591f5b00735" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42413 } }
I20260925 10:08:23.214308 19600 leader_election.cc:288] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cf96a45477f4cb8b065ed6b1ba8914c (127.14.41.67:33867), b9c52203f34d4be5b9bf9591f5b00735 (127.14.41.65:42413)
I20260925 10:08:23.216194 19592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.214608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:46335 (local address 127.14.41.65:42413)
0925 10:08:23.214657 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:23.214659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:23.214688 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:08:23.214799 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:23.214801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:23.214808 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:23.214997 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:23.215715 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:23.215717 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:23.215718 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:23.215724 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:23.215744 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:23.215749 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:23.215988 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:23.216096 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 10:08:23.216114 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:08:23.216194 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.214534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:42413 (local address 127.14.41.66:46335)
0925 10:08:23.214599 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:23.214615 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:23.214688 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:23.215005 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:23.215006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:23.215009 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:23.215577 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:08:23.215578 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:23.215592 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:08:23.215606 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:23.216011 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:23.216012 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:23.216013 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 10:08:23.216094 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 10:08:23.216113 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 10:08:23.216279 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.214683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:50145 (local address 127.14.41.67:33867)
0925 10:08:23.214763 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:23.214765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:23.214949 (+ 184us) server_negotiation.cc:407] Connection header received
0925 10:08:23.215003 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:23.215004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:23.215009 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:23.215062 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:23.216058 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:23.216060 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:23.216060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:23.216064 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:23.216081 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:23.216086 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:23.216139 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:23.216185 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 10:08:23.216198 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:08:23.216501 19603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.214534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:33867 (local address 127.14.41.66:50145)
0925 10:08:23.214876 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:23.214884 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:23.214942 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:23.215064 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:23.215065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:23.215067 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:23.215912 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:08:23.215913 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:23.215920 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:08:23.215979 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:23.216134 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:23.216135 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:23.216136 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 10:08:23.216176 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 10:08:23.216191 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":178,"thread_start_us":105,"threads_started":1}
I20260925 10:08:23.216698 19272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc18273ac5741369d7f109e823f52f5" candidate_uuid: "08bab9891e984d4a823706b0661c440d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9c52203f34d4be5b9bf9591f5b00735" is_pre_election: true
I20260925 10:08:23.216897 19272 raft_consensus.cc:2454] T bcc18273ac5741369d7f109e823f52f5 P b9c52203f34d4be5b9bf9591f5b00735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08bab9891e984d4a823706b0661c440d in term 0.
I20260925 10:08:23.217265 19336 leader_election.cc:302] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bab9891e984d4a823706b0661c440d, b9c52203f34d4be5b9bf9591f5b00735; no voters:
I20260925 10:08:23.217423 19600 raft_consensus.cc:2781] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:23.217496 19600 raft_consensus.cc:482] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:23.217459 19534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc18273ac5741369d7f109e823f52f5" candidate_uuid: "08bab9891e984d4a823706b0661c440d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf96a45477f4cb8b065ed6b1ba8914c" is_pre_election: true
I20260925 10:08:23.217543 19600 raft_consensus.cc:3037] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:23.217629 19534 raft_consensus.cc:2454] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08bab9891e984d4a823706b0661c440d in term 0.
I20260925 10:08:23.218528 19600 raft_consensus.cc:504] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf96a45477f4cb8b065ed6b1ba8914c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33867 } } peers { permanent_uuid: "08bab9891e984d4a823706b0661c440d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45915 } } peers { permanent_uuid: "b9c52203f34d4be5b9bf9591f5b00735" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42413 } }
I20260925 10:08:23.218744 19600 leader_election.cc:288] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [CANDIDATE]: Term 1 election: Requested vote from peers 4cf96a45477f4cb8b065ed6b1ba8914c (127.14.41.67:33867), b9c52203f34d4be5b9bf9591f5b00735 (127.14.41.65:42413)
I20260925 10:08:23.218830 19534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc18273ac5741369d7f109e823f52f5" candidate_uuid: "08bab9891e984d4a823706b0661c440d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf96a45477f4cb8b065ed6b1ba8914c"
I20260925 10:08:23.218919 19534 raft_consensus.cc:3037] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:23.218942 19272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcc18273ac5741369d7f109e823f52f5" candidate_uuid: "08bab9891e984d4a823706b0661c440d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9c52203f34d4be5b9bf9591f5b00735"
I20260925 10:08:23.219023 19272 raft_consensus.cc:3037] T bcc18273ac5741369d7f109e823f52f5 P b9c52203f34d4be5b9bf9591f5b00735 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:23.219703 19534 raft_consensus.cc:2454] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bab9891e984d4a823706b0661c440d in term 1.
I20260925 10:08:23.219720 19272 raft_consensus.cc:2454] T bcc18273ac5741369d7f109e823f52f5 P b9c52203f34d4be5b9bf9591f5b00735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08bab9891e984d4a823706b0661c440d in term 1.
I20260925 10:08:23.219882 19339 leader_election.cc:302] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08bab9891e984d4a823706b0661c440d, 4cf96a45477f4cb8b065ed6b1ba8914c; no voters:
I20260925 10:08:23.220011 19600 raft_consensus.cc:2781] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:23.220186 19600 raft_consensus.cc:686] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [term 1 LEADER]: Becoming Leader. State: Replica: 08bab9891e984d4a823706b0661c440d, State: Running, Role: LEADER
I20260925 10:08:23.220290 19600 consensus_queue.cc:237] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf96a45477f4cb8b065ed6b1ba8914c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33867 } } peers { permanent_uuid: "08bab9891e984d4a823706b0661c440d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45915 } } peers { permanent_uuid: "b9c52203f34d4be5b9bf9591f5b00735" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42413 } }
I20260925 10:08:23.221226 19130 catalog_manager.cc:5495] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d reported cstate change: term changed from 0 to 1, leader changed from <none> to 08bab9891e984d4a823706b0661c440d (127.14.41.66). New cstate: current_term: 1 leader_uuid: "08bab9891e984d4a823706b0661c440d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf96a45477f4cb8b065ed6b1ba8914c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08bab9891e984d4a823706b0661c440d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9c52203f34d4be5b9bf9591f5b00735" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42413 } health_report { overall_health: UNKNOWN } } }
W20260925 10:08:23.225253 19450 tablet.cc:2369] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:08:23.245040 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.243890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55374 (local address 127.14.41.66:45915)
0925 10:08:23.244005 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:23.244006 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:23.244014 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:08:23.244050 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:23.244050 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:23.244063 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:23.244130 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:23.244755 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:23.244757 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:23.244757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:23.244762 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:23.244778 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:23.244782 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:23.244939 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:23.244957 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 10:08:23.244970 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 10:08:23.247052 19534 raft_consensus.cc:1261] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c [term 1 FOLLOWER]: Refusing update from remote peer 08bab9891e984d4a823706b0661c440d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:23.247052 19272 raft_consensus.cc:1261] T bcc18273ac5741369d7f109e823f52f5 P b9c52203f34d4be5b9bf9591f5b00735 [term 1 FOLLOWER]: Refusing update from remote peer 08bab9891e984d4a823706b0661c440d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:23.247466 19600 consensus_queue.cc:1035] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9c52203f34d4be5b9bf9591f5b00735" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:23.247668 19604 consensus_queue.cc:1035] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cf96a45477f4cb8b065ed6b1ba8914c" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 33867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:23.250396 19609 mvcc.cc:205] Tried to move back new op lower bound from 7333195379696553984 to 7333195379591139328. Current Snapshot: MvccSnapshot[applied={T|T < 7333195379696553984}]
I20260925 10:08:23.253304 19130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:23.253451 19130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:23.254276 19130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c829a82082384557a9029323091c0dd1: Sending DeleteTablet for 3 replicas of tablet bcc18273ac5741369d7f109e823f52f5
I20260925 10:08:23.254633 19383 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc18273ac5741369d7f109e823f52f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:23 UTC) from {username='slave'} at 127.0.0.1:55358
I20260925 10:08:23.254647 19514 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc18273ac5741369d7f109e823f52f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:23 UTC) from {username='slave'} at 127.0.0.1:41962
I20260925 10:08:23.254757 19252 tablet_service.cc:1514] Processing DeleteTablet for tablet bcc18273ac5741369d7f109e823f52f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:23 UTC) from {username='slave'} at 127.0.0.1:35300
I20260925 10:08:23.255038 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 19190
I20260925 10:08:23.255033 19616 tablet_replica.cc:330] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d: stopping tablet replica
I20260925 10:08:23.255792 19616 raft_consensus.cc:2229] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:08:23.255990 19616 raft_consensus.cc:2258] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:23.256531 19616 ts_tablet_manager.cc:1905] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:23.258832 19616 ts_tablet_manager.cc:1918] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:08:23.258919 19616 log.cc:1196] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/bcc18273ac5741369d7f109e823f52f5
I20260925 10:08:23.259189 19616 ts_tablet_manager.cc:1939] T bcc18273ac5741369d7f109e823f52f5 P 08bab9891e984d4a823706b0661c440d: Deleting consensus metadata
I20260925 10:08:23.259809 19115 catalog_manager.cc:4841] TS 08bab9891e984d4a823706b0661c440d (127.14.41.66:45915): tablet bcc18273ac5741369d7f109e823f52f5 (table test-table [id=3e16e0d9170649cfb9aa42807f288049]) successfully deleted
W20260925 10:08:23.263504 19114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.41.65:42413 (error 108)
I20260925 10:08:23.264051 19617 tablet_replica.cc:330] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c: stopping tablet replica
I20260925 10:08:23.264115 19588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.263964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:42413 (local address 127.0.0.1:35304)
0925 10:08:23.264023 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:23.264075 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.65:42413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 10:08:23.264130 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 19321
W20260925 10:08:23.264248 19114 catalog_manager.cc:4584] TS b9c52203f34d4be5b9bf9591f5b00735 (127.14.41.65:42413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bcc18273ac5741369d7f109e823f52f5: Network error: Client connection negotiation failed: client connection to 127.14.41.65:42413: connect: Connection refused (error 111)
I20260925 10:08:23.265908 19617 raft_consensus.cc:2229] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:23.266077 19617 raft_consensus.cc:2258] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:23.266822 19617 ts_tablet_manager.cc:1905] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:23.268728 19617 ts_tablet_manager.cc:1918] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:08:23.268829 19617 log.cc:1196] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/bcc18273ac5741369d7f109e823f52f5
I20260925 10:08:23.269116 19617 ts_tablet_manager.cc:1939] T bcc18273ac5741369d7f109e823f52f5 P 4cf96a45477f4cb8b065ed6b1ba8914c: Deleting consensus metadata
I20260925 10:08:23.269757 19117 catalog_manager.cc:4841] TS 4cf96a45477f4cb8b065ed6b1ba8914c (127.14.41.67:33867): tablet bcc18273ac5741369d7f109e823f52f5 (table test-table [id=3e16e0d9170649cfb9aa42807f288049]) successfully deleted
I20260925 10:08:23.272171 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 19452
I20260925 10:08:23.277264 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 19099
2026-09-25T10:08:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (723 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 10:08:23.298295 14501 mini_oidc.cc:152] Starting JWKS server
I20260925 10:08:23.299474 14501 webserver.cc:460] Webserver started at https://127.0.0.1:42663/ using document root <none> and password file <none>
I20260925 10:08:23.299563 14501 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 10:08:23.299710 14501 webserver.cc:460] Webserver started at http://127.0.0.1:41171/ using document root <none> and password file <none>
2026-09-25T10:08:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:23Z Disabled control of system clock
I20260925 10:08:23.308539 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:43061
--webserver_interface=127.14.41.126
--webserver_port=0
--builtin_ntp_servers=127.14.41.84:41951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:43061
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42663/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330877480195-14501-0/testchainca.cert with env {}
W20260925 10:08:23.391868 19625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:23.392182 19625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:23.392226 19625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:23.393594 19625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:23.393682 19625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:23.393702 19625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:23.393724 19625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:23.393756 19625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:08:23.393785 19625 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:08:23.393805 19625 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42663/jwks.json
I20260925 10:08:23.395571 19625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:41951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:43061
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:43061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42663/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330877480195-14501-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:23.395895 19625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:23.396144 19625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:23.401973 19630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:23.401970 19631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:23.402088 19633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:23.402504 19625 server_base.cc:1034] running on GCE node
I20260925 10:08:23.402786 19625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:23.403126 19625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:23.404296 19625 hybrid_clock.cc:648] HybridClock initialized: now 1790330903404268 us; error 44 us; skew 500 ppm
I20260925 10:08:23.405776 19625 webserver.cc:460] Webserver started at http://127.14.41.126:41739/ using document root <none> and password file <none>
I20260925 10:08:23.406065 19625 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:23.406141 19625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:23.406298 19625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:23.407421 19625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "320a22fdce8c44309d612600a12c186a"
format_stamp: "Formatted at 2026-09-25 10:08:23 on dist-test-slave-s6tm"
server_key: "672c3053535e3f93a6525439d674c412"
server_key_iv: "cd553d91eb9a4bc76c2296748cbbb2a9"
server_key_version: "encryptionkey@0"
I20260925 10:08:23.407814 19625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "320a22fdce8c44309d612600a12c186a"
format_stamp: "Formatted at 2026-09-25 10:08:23 on dist-test-slave-s6tm"
server_key: "672c3053535e3f93a6525439d674c412"
server_key_iv: "cd553d91eb9a4bc76c2296748cbbb2a9"
server_key_version: "encryptionkey@0"
I20260925 10:08:23.409317 19625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:23.410284 19639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:23.410575 19625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:23.410686 19625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "320a22fdce8c44309d612600a12c186a"
format_stamp: "Formatted at 2026-09-25 10:08:23 on dist-test-slave-s6tm"
server_key: "672c3053535e3f93a6525439d674c412"
server_key_iv: "cd553d91eb9a4bc76c2296748cbbb2a9"
server_key_version: "encryptionkey@0"
I20260925 10:08:23.410763 19625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:23.430611 19625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:23.430996 19625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:23.431367 19625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:23.436312 19625 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:43061
I20260925 10:08:23.436411 19691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:43061 every 8 connection(s)
I20260925 10:08:23.436802 19625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:23.437835 19692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:23.440456 19692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a: Bootstrap starting.
I20260925 10:08:23.441100 19692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:23.441434 19692 log.cc:824] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:23.442181 19692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a: No bootstrap required, opened a new log
I20260925 10:08:23.443670 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 19625
I20260925 10:08:23.443761 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:08:23.443989 14501 external_mini_cluster.cc:1428] Setting key 4d061a79797415b98c787e13fc5eee38
I20260925 10:08:23.443995 19692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a22fdce8c44309d612600a12c186a" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43061 } }
I20260925 10:08:23.444214 19692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:23.444263 19692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320a22fdce8c44309d612600a12c186a, State: Initialized, Role: FOLLOWER
I20260925 10:08:23.444415 19692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a22fdce8c44309d612600a12c186a" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43061 } }
I20260925 10:08:23.444501 19692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:08:23.444538 19692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:23.444597 19692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:23.445173 19692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a22fdce8c44309d612600a12c186a" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43061 } }
I20260925 10:08:23.445308 19692 leader_election.cc:302] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 320a22fdce8c44309d612600a12c186a; no voters:
I20260925 10:08:23.445515 19692 leader_election.cc:288] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:08:23.445577 19697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:23.445724 19697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a [term 1 LEADER]: Becoming Leader. State: Replica: 320a22fdce8c44309d612600a12c186a, State: Running, Role: LEADER
I20260925 10:08:23.445798 19692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:23.445833 19697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a22fdce8c44309d612600a12c186a" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43061 } }
I20260925 10:08:23.446197 19697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 320a22fdce8c44309d612600a12c186a. Latest consensus state: current_term: 1 leader_uuid: "320a22fdce8c44309d612600a12c186a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a22fdce8c44309d612600a12c186a" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43061 } } }
I20260925 10:08:23.446274 19697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:23.446235 19698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "320a22fdce8c44309d612600a12c186a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "320a22fdce8c44309d612600a12c186a" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43061 } } }
I20260925 10:08:23.446434 19698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a [sys.catalog]: This master's current role is: LEADER
W20260925 10:08:23.447613 19712 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:08:23.447713 19712 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:08:23.448019 19703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:23.448226 19703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:23.450150 19703 catalog_manager.cc:1347] Generated new cluster ID: 577d17d08166405d863d1de27bb3b8ba
I20260925 10:08:23.450206 19703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:08:23.450543 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.444472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40700 (local address 127.14.41.126:43061)
0925 10:08:23.444767 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:23.444777 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:23.444818 (+ 41us) server_negotiation.cc:407] Connection header received
0925 10:08:23.444995 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:23.445002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:23.445069 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:23.445227 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:23.445452 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:23.447195 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:23.448415 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:23.448824 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:23.448967 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:23.449732 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:23.449737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:23.449737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:23.449953 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:23.449999 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:23.450014 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:23.450088 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:23.450176 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 10:08:23.450364 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":93,"threads_started":1}
I20260925 10:08:23.451483 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:08:23.462255 19703 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:08:23.462831 19703 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:23.466591 19703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 320a22fdce8c44309d612600a12c186a: Generated new TSK 0
I20260925 10:08:23.466827 19703 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 10:08:23.493304 19722 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 10:08:23.493432 19722 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 10:08:23.493692 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.477285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40708 (local address 127.14.41.126:43061)
0925 10:08:23.477338 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:23.477340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:23.477484 (+ 144us) server_negotiation.cc:407] Connection header received
0925 10:08:23.477565 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:23.477566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:23.477579 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:23.477673 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:08:23.477825 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:23.478654 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:23.479627 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:23.479965 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:23.480039 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:23.480283 (+ 244us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:08:23.493448 (+ 13165us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42663/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 10:08:23.493616 (+ 168us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40708: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42663/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":28}
W20260925 10:08:23.493760 19696 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40708: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42663/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 10:08:23.494148 19721 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:23.477183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43061 (local address 127.0.0.1:40708)
0925 10:08:23.477396 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:23.477407 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:23.477478 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:23.477674 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:23.477675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:23.477681 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:08:23.477742 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:23.477744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:23.478883 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:23.478884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:23.479520 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:23.479523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:23.480061 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:23.480062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:23.480113 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:23.480139 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:08:23.493647 (+ 13508us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42663/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 10:08:23.493785 (+ 138us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.41.126:43061: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42663/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.14.41.126:43061: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42663/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 10:08:23.494685 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 19625
2026-09-25T10:08:23Z chronyd exiting
I20260925 10:08:23.531338 14501 test_util.cc:175] -----------------------------------------------
I20260925 10:08:23.531428 14501 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790330877480195-14501-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (234 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 10:08:23.532580 14501 mini_oidc.cc:152] Starting JWKS server
I20260925 10:08:23.533397 14501 webserver.cc:460] Webserver started at https://127.0.0.1:38855/ using document root <none> and password file <none>
I20260925 10:08:23.533454 14501 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 10:08:23.533641 14501 webserver.cc:460] Webserver started at http://127.0.0.1:44421/ using document root <none> and password file <none>
2026-09-25T10:08:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:23Z Disabled control of system clock
I20260925 10:08:23.542415 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:40739
--webserver_interface=127.14.41.126
--webserver_port=0
--builtin_ntp_servers=127.14.41.84:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:40739
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38855/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330877480195-14501-0/testchainca.cert with env {}
W20260925 10:08:23.624573 19728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:23.624805 19728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:23.624830 19728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:23.626063 19728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:23.626109 19728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:23.626122 19728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:23.626135 19728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:23.626147 19728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:08:23.626204 19728 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:08:23.626219 19728 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38855/jwks.json
I20260925 10:08:23.627843 19728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:40739
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:40739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38855/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330877480195-14501-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:23.628170 19728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:23.628405 19728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:23.633443 19733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:23.633563 19728 server_base.cc:1034] running on GCE node
W20260925 10:08:23.633458 19736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:23.633462 19734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:23.634027 19728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:23.634351 19728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:23.635555 19728 hybrid_clock.cc:648] HybridClock initialized: now 1790330903635537 us; error 27 us; skew 500 ppm
I20260925 10:08:23.637024 19728 webserver.cc:460] Webserver started at http://127.14.41.126:41447/ using document root <none> and password file <none>
I20260925 10:08:23.637270 19728 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:23.637341 19728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:23.637508 19728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:23.638716 19728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "0d8ff1836b6745e2ba3acc4c76b233c0"
format_stamp: "Formatted at 2026-09-25 10:08:23 on dist-test-slave-s6tm"
server_key: "cb6d063251170280255dd7af24bdd01d"
server_key_iv: "87c3ac5d31b48a7a6e9c356bfcf7e65d"
server_key_version: "encryptionkey@0"
I20260925 10:08:23.639102 19728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "0d8ff1836b6745e2ba3acc4c76b233c0"
format_stamp: "Formatted at 2026-09-25 10:08:23 on dist-test-slave-s6tm"
server_key: "cb6d063251170280255dd7af24bdd01d"
server_key_iv: "87c3ac5d31b48a7a6e9c356bfcf7e65d"
server_key_version: "encryptionkey@0"
I20260925 10:08:23.640575 19728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 10:08:23.641399 19742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:23.641628 19728 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:23.641737 19728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "0d8ff1836b6745e2ba3acc4c76b233c0"
format_stamp: "Formatted at 2026-09-25 10:08:23 on dist-test-slave-s6tm"
server_key: "cb6d063251170280255dd7af24bdd01d"
server_key_iv: "87c3ac5d31b48a7a6e9c356bfcf7e65d"
server_key_version: "encryptionkey@0"
I20260925 10:08:23.641816 19728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:23.660893 19728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:23.661278 19728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:23.661453 19728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:23.665663 19728 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:40739
I20260925 10:08:23.665701 19794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:40739 every 8 connection(s)
I20260925 10:08:23.666107 19728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:23.666847 19795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:23.667307 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 19728
I20260925 10:08:23.667399 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:08:23.667610 14501 external_mini_cluster.cc:1428] Setting key e1472c187b3d28aa0f77fd850e97fa37
I20260925 10:08:23.671571 19795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0: Bootstrap starting.
I20260925 10:08:23.672327 19795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:23.672634 19795 log.cc:824] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:23.673512 19795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0: No bootstrap required, opened a new log
I20260925 10:08:23.674396 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.668460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34620 (local address 127.14.41.126:40739)
0925 10:08:23.668656 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:23.668665 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:23.668691 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:08:23.669268 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:23.669277 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:23.669348 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:23.669507 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:23.669961 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:23.670949 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:23.672145 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:23.672664 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:23.672846 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:23.673617 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:23.673621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:23.673622 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:23.673834 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:23.673876 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:23.673893 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:23.673963 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:23.674068 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 10:08:23.674160 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260925 10:08:23.675580 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:08:23.676203 19795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8ff1836b6745e2ba3acc4c76b233c0" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 40739 } }
I20260925 10:08:23.676429 19795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:23.676481 19795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8ff1836b6745e2ba3acc4c76b233c0, State: Initialized, Role: FOLLOWER
I20260925 10:08:23.676605 19795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8ff1836b6745e2ba3acc4c76b233c0" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 40739 } }
I20260925 10:08:23.676687 19795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:08:23.676733 19795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:23.676792 19795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:23.677361 19795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8ff1836b6745e2ba3acc4c76b233c0" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 40739 } }
I20260925 10:08:23.677443 19795 leader_election.cc:302] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d8ff1836b6745e2ba3acc4c76b233c0; no voters:
I20260925 10:08:23.677574 19795 leader_election.cc:288] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:08:23.677657 19800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:23.677838 19795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:23.677925 19800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0 [term 1 LEADER]: Becoming Leader. State: Replica: 0d8ff1836b6745e2ba3acc4c76b233c0, State: Running, Role: LEADER
I20260925 10:08:23.678035 19800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8ff1836b6745e2ba3acc4c76b233c0" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 40739 } }
I20260925 10:08:23.678447 19802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d8ff1836b6745e2ba3acc4c76b233c0. Latest consensus state: current_term: 1 leader_uuid: "0d8ff1836b6745e2ba3acc4c76b233c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8ff1836b6745e2ba3acc4c76b233c0" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 40739 } } }
I20260925 10:08:23.678454 19801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d8ff1836b6745e2ba3acc4c76b233c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8ff1836b6745e2ba3acc4c76b233c0" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 40739 } } }
I20260925 10:08:23.678555 19802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:23.678558 19801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:23.678757 19809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:23.679312 19809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:23.681002 19809 catalog_manager.cc:1347] Generated new cluster ID: 6850ce226a7c4ecb86c1f39fa167b67f
I20260925 10:08:23.681053 19809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:08:23.694114 19809 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:08:23.694689 19809 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:23.702904 19809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d8ff1836b6745e2ba3acc4c76b233c0: Generated new TSK 0
I20260925 10:08:23.703145 19809 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 10:08:23.738750 19825 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 10:08:23.738857 19825 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 10:08:23.739324 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.715454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34636 (local address 127.14.41.126:40739)
0925 10:08:23.715533 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:23.715535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:23.715635 (+ 100us) server_negotiation.cc:407] Connection header received
0925 10:08:23.715743 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:23.715744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:23.715758 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:23.715860 (+ 102us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:08:23.716066 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:23.717060 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:23.718225 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:23.718598 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:23.718676 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:23.718890 (+ 214us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:08:23.739006 (+ 20116us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 10:08:23.739210 (+ 204us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34636: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 10:08:23.739393 19824 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:23.715339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:40739 (local address 127.0.0.1:34636)
0925 10:08:23.715574 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:23.715587 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:23.715653 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:23.715870 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:23.715872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:23.715874 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:08:23.715973 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:23.715976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:23.717159 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:23.717160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:23.718128 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:23.718132 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:23.718679 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:23.718680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:23.718744 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:23.718749 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:08:23.739150 (+ 20401us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 10:08:23.739279 (+ 129us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.41.126:40739: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":85,"threads_started":1}
W20260925 10:08:23.739400 19798 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34636: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38855/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260925 10:08:23.739668 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 19728
2026-09-25T10:08:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (245 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 10:08:23.778128 14501 mini_oidc.cc:152] Starting JWKS server
I20260925 10:08:23.778848 14501 webserver.cc:460] Webserver started at https://127.0.0.1:42767/ using document root <none> and password file <none>
I20260925 10:08:23.778927 14501 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 10:08:23.779112 14501 webserver.cc:460] Webserver started at http://127.0.0.1:45649/ using document root <none> and password file <none>
2026-09-25T10:08:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:23Z Disabled control of system clock
I20260925 10:08:23.787148 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:45175
--webserver_interface=127.14.41.126
--webserver_port=0
--builtin_ntp_servers=127.14.41.84:40315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:45175
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42767/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 10:08:23.873458 19831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:23.873725 19831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:23.873767 19831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:23.875098 19831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:23.875181 19831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:23.875214 19831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:23.875243 19831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:23.875274 19831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:08:23.875296 19831 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:08:23.875321 19831 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42767/jwks.json
I20260925 10:08:23.876825 19831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:45175
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:45175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42767/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:23.877130 19831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:23.877472 19831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:23.883463 19837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:23.883476 19839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:23.883528 19831 server_base.cc:1034] running on GCE node
W20260925 10:08:23.883451 19836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:23.884110 19831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:23.884495 19831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:23.885663 19831 hybrid_clock.cc:648] HybridClock initialized: now 1790330903885652 us; error 36 us; skew 500 ppm
I20260925 10:08:23.887233 19831 webserver.cc:460] Webserver started at http://127.14.41.126:33661/ using document root <none> and password file <none>
I20260925 10:08:23.887495 19831 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:23.887545 19831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:23.887720 19831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:23.889286 19831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "e59772dfb05f41cd8ef8d53e214074b6"
format_stamp: "Formatted at 2026-09-25 10:08:23 on dist-test-slave-s6tm"
server_key: "1ca5a37b111c411c4113394ac7dd8929"
server_key_iv: "1b70804579fe13e17bbbed3aed3d6b2f"
server_key_version: "encryptionkey@0"
I20260925 10:08:23.889705 19831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "e59772dfb05f41cd8ef8d53e214074b6"
format_stamp: "Formatted at 2026-09-25 10:08:23 on dist-test-slave-s6tm"
server_key: "1ca5a37b111c411c4113394ac7dd8929"
server_key_iv: "1b70804579fe13e17bbbed3aed3d6b2f"
server_key_version: "encryptionkey@0"
I20260925 10:08:23.891201 19831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:08:23.892174 19845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:23.892412 19831 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:23.892524 19831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "e59772dfb05f41cd8ef8d53e214074b6"
format_stamp: "Formatted at 2026-09-25 10:08:23 on dist-test-slave-s6tm"
server_key: "1ca5a37b111c411c4113394ac7dd8929"
server_key_iv: "1b70804579fe13e17bbbed3aed3d6b2f"
server_key_version: "encryptionkey@0"
I20260925 10:08:23.892622 19831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:23.902408 19831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:23.902753 19831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:23.902916 19831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:23.906983 19831 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:45175
I20260925 10:08:23.907090 19897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:45175 every 8 connection(s)
I20260925 10:08:23.907464 19831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:23.908440 19898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:23.911343 19898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6: Bootstrap starting.
I20260925 10:08:23.912017 19898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:23.912310 19898 log.cc:824] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:23.912318 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 19831
I20260925 10:08:23.912446 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:08:23.912643 14501 external_mini_cluster.cc:1428] Setting key 368f89513b366b366b391360edf7a303
I20260925 10:08:23.913102 19898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6: No bootstrap required, opened a new log
I20260925 10:08:23.915915 19898 raft_consensus.cc:348] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59772dfb05f41cd8ef8d53e214074b6" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45175 } }
I20260925 10:08:23.916119 19898 raft_consensus.cc:374] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:23.916149 19898 raft_consensus.cc:729] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59772dfb05f41cd8ef8d53e214074b6, State: Initialized, Role: FOLLOWER
I20260925 10:08:23.916272 19898 consensus_queue.cc:260] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59772dfb05f41cd8ef8d53e214074b6" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45175 } }
I20260925 10:08:23.916347 19898 raft_consensus.cc:388] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:08:23.916406 19898 raft_consensus.cc:482] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:23.916465 19898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:23.917049 19898 raft_consensus.cc:504] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59772dfb05f41cd8ef8d53e214074b6" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45175 } }
I20260925 10:08:23.917176 19898 leader_election.cc:302] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e59772dfb05f41cd8ef8d53e214074b6; no voters:
I20260925 10:08:23.917359 19898 leader_election.cc:288] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:08:23.917398 19903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:23.917512 19903 raft_consensus.cc:686] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6 [term 1 LEADER]: Becoming Leader. State: Replica: e59772dfb05f41cd8ef8d53e214074b6, State: Running, Role: LEADER
I20260925 10:08:23.917606 19903 consensus_queue.cc:237] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59772dfb05f41cd8ef8d53e214074b6" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45175 } }
I20260925 10:08:23.917632 19898 sys_catalog.cc:564] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:23.917966 19903 sys_catalog.cc:455] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e59772dfb05f41cd8ef8d53e214074b6. Latest consensus state: current_term: 1 leader_uuid: "e59772dfb05f41cd8ef8d53e214074b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59772dfb05f41cd8ef8d53e214074b6" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45175 } } }
I20260925 10:08:23.918212 19903 sys_catalog.cc:458] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:23.918462 19907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:23.917975 19905 sys_catalog.cc:455] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e59772dfb05f41cd8ef8d53e214074b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59772dfb05f41cd8ef8d53e214074b6" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45175 } } }
I20260925 10:08:23.918658 19905 sys_catalog.cc:458] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:23.919795 19907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:23.920008 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.913183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48680 (local address 127.14.41.126:45175)
0925 10:08:23.913410 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:23.913419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:23.913455 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:08:23.914406 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:23.914416 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:23.914495 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:23.914650 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:23.914869 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:23.916320 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:23.917211 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:23.918018 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:23.918203 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:23.919110 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:23.919114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:23.919115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:23.919347 (+ 232us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:23.919398 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:23.919416 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:23.919494 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:23.919632 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 10:08:23.919719 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
I20260925 10:08:23.921017 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:08:23.925295 19907 catalog_manager.cc:1347] Generated new cluster ID: 004221aa8a134370acaeaef622470833
I20260925 10:08:23.925388 19907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:08:23.932603 19907 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:08:23.933175 19907 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:23.937711 19907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e59772dfb05f41cd8ef8d53e214074b6: Generated new TSK 0
I20260925 10:08:23.937929 19907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:08:23.951072 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:23.946928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.14.41.126:45175)
0925 10:08:23.946997 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:23.947000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:23.947162 (+ 162us) server_negotiation.cc:407] Connection header received
0925 10:08:23.947245 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:23.947246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:23.947258 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:23.947322 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:08:23.947511 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:23.948111 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:23.949284 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:23.949511 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:23.949567 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:23.949771 (+ 204us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:08:23.950843 (+ 1072us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42767/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 10:08:23.950990 (+ 147us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48684: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42767/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 10:08:23.951143 19902 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48684: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42767/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 10:08:23.951540 19927 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:23.946856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:45175 (local address 127.0.0.1:48684)
0925 10:08:23.947082 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:23.947094 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:23.947168 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:23.947367 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:23.947368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:23.947371 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:08:23.947433 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:23.947436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:23.948194 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:23.948195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:23.949194 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:23.949197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:23.949595 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:23.949597 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:23.949660 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:23.949667 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:08:23.951315 (+ 1648us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42767/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 10:08:23.951434 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.41.126:45175: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42767/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":156,"threads_started":1}
I20260925 10:08:23.951769 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 19831
E20260925 10:08:23.962843 19928 webserver.cc:533] Webserver: sslize failed:
E20260925 10:08:23.962929 19928 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T10:08:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (212 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T10:08:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:23Z Disabled control of system clock
I20260925 10:08:23.999194 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:33259
--webserver_interface=127.14.41.126
--webserver_port=0
--builtin_ntp_servers=127.14.41.84:38595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:33259
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330877480195-14501-0/testchainca.cert with env {}
W20260925 10:08:24.085727 19932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:24.085983 19932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:24.086021 19932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:24.087201 19932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:24.087271 19932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:24.087303 19932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:24.087332 19932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:24.087364 19932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:08:24.087386 19932 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:08:24.087407 19932 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 10:08:24.088928 19932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:38595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:33259
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:33259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330877480195-14501-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:24.089219 19932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:24.089581 19932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:24.095418 19938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:24.095450 19932 server_base.cc:1034] running on GCE node
W20260925 10:08:24.095448 19937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:24.095418 19940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:24.096060 19932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:24.096408 19932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:24.097581 19932 hybrid_clock.cc:648] HybridClock initialized: now 1790330904097555 us; error 43 us; skew 500 ppm
I20260925 10:08:24.098945 19932 webserver.cc:460] Webserver started at http://127.14.41.126:39197/ using document root <none> and password file <none>
I20260925 10:08:24.099196 19932 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:24.099247 19932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:24.099466 19932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:24.100694 19932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "bccbfa507bad42ee956fc0354aeb970f"
format_stamp: "Formatted at 2026-09-25 10:08:24 on dist-test-slave-s6tm"
server_key: "eb4f0404dd15d2bbed7245c5204c4ab2"
server_key_iv: "dadd6ef7d501e02cdf389fce0f446e3f"
server_key_version: "encryptionkey@0"
I20260925 10:08:24.101085 19932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "bccbfa507bad42ee956fc0354aeb970f"
format_stamp: "Formatted at 2026-09-25 10:08:24 on dist-test-slave-s6tm"
server_key: "eb4f0404dd15d2bbed7245c5204c4ab2"
server_key_iv: "dadd6ef7d501e02cdf389fce0f446e3f"
server_key_version: "encryptionkey@0"
I20260925 10:08:24.102553 19932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:08:24.103370 19946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:24.103564 19932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:24.103670 19932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "bccbfa507bad42ee956fc0354aeb970f"
format_stamp: "Formatted at 2026-09-25 10:08:24 on dist-test-slave-s6tm"
server_key: "eb4f0404dd15d2bbed7245c5204c4ab2"
server_key_iv: "dadd6ef7d501e02cdf389fce0f446e3f"
server_key_version: "encryptionkey@0"
I20260925 10:08:24.103753 19932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:24.129928 19932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:24.130292 19932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:24.130470 19932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:24.134714 19932 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:33259
I20260925 10:08:24.134853 19998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:33259 every 8 connection(s)
I20260925 10:08:24.135195 19932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:24.136387 19999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:24.138911 19999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f: Bootstrap starting.
I20260925 10:08:24.139561 19999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:24.139897 19999 log.cc:824] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:24.140868 19999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f: No bootstrap required, opened a new log
I20260925 10:08:24.142601 19999 raft_consensus.cc:348] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccbfa507bad42ee956fc0354aeb970f" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 33259 } }
I20260925 10:08:24.142759 19999 raft_consensus.cc:374] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:24.142784 19999 raft_consensus.cc:729] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bccbfa507bad42ee956fc0354aeb970f, State: Initialized, Role: FOLLOWER
I20260925 10:08:24.142957 19999 consensus_queue.cc:260] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccbfa507bad42ee956fc0354aeb970f" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 33259 } }
I20260925 10:08:24.143023 19999 raft_consensus.cc:388] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:08:24.143045 19999 raft_consensus.cc:482] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:24.143116 19999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:24.143703 19999 raft_consensus.cc:504] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccbfa507bad42ee956fc0354aeb970f" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 33259 } }
I20260925 10:08:24.143838 19999 leader_election.cc:302] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bccbfa507bad42ee956fc0354aeb970f; no voters:
I20260925 10:08:24.144053 19999 leader_election.cc:288] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:08:24.144141 20002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:24.144313 20002 raft_consensus.cc:686] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f [term 1 LEADER]: Becoming Leader. State: Replica: bccbfa507bad42ee956fc0354aeb970f, State: Running, Role: LEADER
I20260925 10:08:24.144351 19999 sys_catalog.cc:564] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:24.144433 20002 consensus_queue.cc:237] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccbfa507bad42ee956fc0354aeb970f" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 33259 } }
I20260925 10:08:24.144548 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 19932
I20260925 10:08:24.144650 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:08:24.144874 14501 external_mini_cluster.cc:1428] Setting key c1652e2ef73ff891c7586fef0a666098
I20260925 10:08:24.144960 20003 sys_catalog.cc:455] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bccbfa507bad42ee956fc0354aeb970f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccbfa507bad42ee956fc0354aeb970f" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 33259 } } }
I20260925 10:08:24.145099 20003 sys_catalog.cc:458] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:24.145468 20011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:24.145802 20004 sys_catalog.cc:455] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bccbfa507bad42ee956fc0354aeb970f. Latest consensus state: current_term: 1 leader_uuid: "bccbfa507bad42ee956fc0354aeb970f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bccbfa507bad42ee956fc0354aeb970f" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 33259 } } }
I20260925 10:08:24.145929 20004 sys_catalog.cc:458] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:24.146027 20011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:24.147652 20011 catalog_manager.cc:1347] Generated new cluster ID: d3ded1a615484e0193534706474260be
I20260925 10:08:24.147710 20011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:08:24.151762 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:24.145576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42006 (local address 127.14.41.126:33259)
0925 10:08:24.145768 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:24.145778 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:24.146049 (+ 271us) server_negotiation.cc:407] Connection header received
0925 10:08:24.146294 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:24.146302 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:24.146388 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:24.146502 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:24.146877 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:24.148332 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:24.149442 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:24.150119 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:24.150255 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:24.151006 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:24.151011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:24.151011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:24.151234 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:24.151277 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:24.151292 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:24.151364 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:24.151434 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 10:08:24.151524 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":61,"threads_started":1}
I20260925 10:08:24.152794 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:08:24.168457 20011 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:08:24.169066 20011 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:24.174445 20011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bccbfa507bad42ee956fc0354aeb970f: Generated new TSK 0
I20260925 10:08:24.174672 20011 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:08:24.610105 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:24.180898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42014 (local address 127.14.41.126:33259)
0925 10:08:24.180963 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:24.180966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:24.181081 (+ 115us) server_negotiation.cc:407] Connection header received
0925 10:08:24.181172 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:24.181173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:24.181183 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:24.181243 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:08:24.181413 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:24.181906 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:24.183035 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:24.183221 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:24.183279 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:24.183477 (+ 198us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:08:24.609827 (+426350us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 10:08:24.610036 (+ 209us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42014: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 10:08:24.610231 20019 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42014: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 10:08:24.610236 20028 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:24.180829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:33259 (local address 127.0.0.1:42014)
0925 10:08:24.181019 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:24.181031 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:24.181100 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:24.181263 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:24.181264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:24.181267 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:08:24.181329 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:24.181332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:24.181984 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:24.181985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:24.182939 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:24.182942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:24.183307 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:24.183308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:24.183374 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:24.183380 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:08:24.610020 (+426640us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 10:08:24.610158 (+ 138us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.14.41.126:33259: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":119,"threads_started":1}
I20260925 10:08:24.610646 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 19932
2026-09-25T10:08:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (638 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (96 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (91 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:08:24 dist-test-slave-s6tm krb5kdc[20040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:08:24 dist-test-slave-s6tm krb5kdc[20040](info): set up 2 sockets
Sep 25 10:08:24 dist-test-slave-s6tm krb5kdc[20040](info): commencing operation
krb5kdc: starting...
W20260925 10:08:25.853710 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:08:25 dist-test-slave-s6tm krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:08:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:08:25.988063 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:40387
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:40387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:26.073643 20053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:26.073916 20053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:26.073944 20053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:26.075304 20053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:26.075362 20053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:26.075377 20053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:26.075392 20053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:26.075403 20053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:08:26.077170 20053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:40387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:40387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:26.077458 20053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:26.077713 20053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:26.083155 20061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:26.083376 20053 server_base.cc:1034] running on GCE node
W20260925 10:08:26.083313 20058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:26.083149 20059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:26.083823 20053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:26.084180 20053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:26.085381 20053 hybrid_clock.cc:648] HybridClock initialized: now 1790330906085366 us; error 50 us; skew 500 ppm
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:26.088490 20053 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:08:26.089023 20053 webserver.cc:460] Webserver started at http://127.14.41.126:45229/ using document root <none> and password file <none>
I20260925 10:08:26.089320 20053 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:26.089373 20053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:26.089529 20053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:26.090754 20053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "d261273b6a4043c3b697a95ad2e789eb"
format_stamp: "Formatted at 2026-09-25 10:08:26 on dist-test-slave-s6tm"
server_key: "ed98942d82293d3646665a1b4df4c6a4"
server_key_iv: "7cd3ddb97ba97c9941165f2a849e8fff"
server_key_version: "encryptionkey@0"
I20260925 10:08:26.091147 20053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "d261273b6a4043c3b697a95ad2e789eb"
format_stamp: "Formatted at 2026-09-25 10:08:26 on dist-test-slave-s6tm"
server_key: "ed98942d82293d3646665a1b4df4c6a4"
server_key_iv: "7cd3ddb97ba97c9941165f2a849e8fff"
server_key_version: "encryptionkey@0"
I20260925 10:08:26.092830 20053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:08:26.093704 20068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:26.093922 20053 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:26.094007 20053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "d261273b6a4043c3b697a95ad2e789eb"
format_stamp: "Formatted at 2026-09-25 10:08:26 on dist-test-slave-s6tm"
server_key: "ed98942d82293d3646665a1b4df4c6a4"
server_key_iv: "7cd3ddb97ba97c9941165f2a849e8fff"
server_key_version: "encryptionkey@0"
I20260925 10:08:26.094069 20053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:26.109802 20053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:26.112311 20053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:26.112435 20053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:26.116380 20053 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:40387
I20260925 10:08:26.116451 20120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:40387 every 8 connection(s)
I20260925 10:08:26.116837 20053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:26.117835 20121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:26.120357 20121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb: Bootstrap starting.
I20260925 10:08:26.120977 20121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:26.121280 20121 log.cc:824] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:26.122135 20121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb: No bootstrap required, opened a new log
I20260925 10:08:26.123590 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 20053
I20260925 10:08:26.123685 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:08:26.124003 14501 external_mini_cluster.cc:1428] Setting key c7b2be07a803171c6c4c703167deec8e
I20260925 10:08:26.124025 20121 raft_consensus.cc:348] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d261273b6a4043c3b697a95ad2e789eb" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 40387 } }
I20260925 10:08:26.124253 20121 raft_consensus.cc:374] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:26.124305 20121 raft_consensus.cc:729] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d261273b6a4043c3b697a95ad2e789eb, State: Initialized, Role: FOLLOWER
I20260925 10:08:26.124471 20121 consensus_queue.cc:260] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d261273b6a4043c3b697a95ad2e789eb" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 40387 } }
I20260925 10:08:26.124656 20121 raft_consensus.cc:388] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:08:26.124737 20121 raft_consensus.cc:482] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:26.124843 20121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:26.126008 20121 raft_consensus.cc:504] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d261273b6a4043c3b697a95ad2e789eb" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 40387 } }
I20260925 10:08:26.126118 20121 leader_election.cc:302] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d261273b6a4043c3b697a95ad2e789eb; no voters:
I20260925 10:08:26.126263 20121 leader_election.cc:288] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:08:26.126353 20126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:26.126534 20126 raft_consensus.cc:686] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb [term 1 LEADER]: Becoming Leader. State: Replica: d261273b6a4043c3b697a95ad2e789eb, State: Running, Role: LEADER
I20260925 10:08:26.126540 20121 sys_catalog.cc:564] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:26.126672 20126 consensus_queue.cc:237] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d261273b6a4043c3b697a95ad2e789eb" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 40387 } }
I20260925 10:08:26.127238 20127 sys_catalog.cc:455] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d261273b6a4043c3b697a95ad2e789eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d261273b6a4043c3b697a95ad2e789eb" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 40387 } } }
I20260925 10:08:26.127326 20127 sys_catalog.cc:458] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:26.127671 20128 sys_catalog.cc:455] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d261273b6a4043c3b697a95ad2e789eb. Latest consensus state: current_term: 1 leader_uuid: "d261273b6a4043c3b697a95ad2e789eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d261273b6a4043c3b697a95ad2e789eb" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 40387 } } }
I20260925 10:08:26.127727 20128 sys_catalog.cc:458] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb [sys.catalog]: This master's current role is: LEADER
W20260925 10:08:26.128537 20142 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:08:26.128630 20142 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:08:26.129356 20135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:26.129560 20135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:26.131376 20135 catalog_manager.cc:1347] Generated new cluster ID: 5dbced30329a4434a97e35a05a669ea8
I20260925 10:08:26.131474 20135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:26.136924 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:26.124532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49584 (local address 127.14.41.126:40387)
0925 10:08:26.124770 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:26.124781 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:26.124846 (+ 65us) server_negotiation.cc:407] Connection header received
0925 10:08:26.124984 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:26.125001 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:26.125062 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:26.125230 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:26.125870 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:26.127344 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:26.128585 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:26.129085 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:26.129242 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:26.132171 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:26.132176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:26.132176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:26.132189 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:26.135094 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:26.135534 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:26.135537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:26.135538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:26.135661 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:26.135900 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:26.135901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:26.135901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:26.136292 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:26.136456 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:26.136550 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 10:08:26.136693 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":126,"threads_started":1}
I20260925 10:08:26.141574 20135 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:08:26.142114 20135 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:26.151768 20135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d261273b6a4043c3b697a95ad2e789eb: Generated new TSK 0
I20260925 10:08:26.152036 20135 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
I20260925 10:08:26.186461 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:40387
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:26.271265 20147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:26.271486 20147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:26.271513 20147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:26.272728 20147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:26.272770 20147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:26.272797 20147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:08:26.274276 20147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:40387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:26.274488 20147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:26.274667 20147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:26.279886 20155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:26.279945 20152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:26.280009 20153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:26.280184 20147 server_base.cc:1034] running on GCE node
I20260925 10:08:26.280417 20147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:26.280664 20147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:26.281859 20147 hybrid_clock.cc:648] HybridClock initialized: now 1790330906281851 us; error 39 us; skew 500 ppm
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:26.284600 20147 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:08:26.285066 20147 webserver.cc:460] Webserver started at http://127.14.41.65:33913/ using document root <none> and password file <none>
I20260925 10:08:26.285295 20147 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:26.285341 20147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:26.285506 20147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:26.286654 20147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "60518bf6cbed481688fff5a5c235155f"
format_stamp: "Formatted at 2026-09-25 10:08:26 on dist-test-slave-s6tm"
server_key: "2318f0d4eb928895446d5bd7222a0548"
server_key_iv: "b369e0eee1152cadfa386ed63d76d290"
server_key_version: "encryptionkey@0"
I20260925 10:08:26.287063 20147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "60518bf6cbed481688fff5a5c235155f"
format_stamp: "Formatted at 2026-09-25 10:08:26 on dist-test-slave-s6tm"
server_key: "2318f0d4eb928895446d5bd7222a0548"
server_key_iv: "b369e0eee1152cadfa386ed63d76d290"
server_key_version: "encryptionkey@0"
I20260925 10:08:26.288525 20147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:08:26.289389 20162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:26.289569 20147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:26.289678 20147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "60518bf6cbed481688fff5a5c235155f"
format_stamp: "Formatted at 2026-09-25 10:08:26 on dist-test-slave-s6tm"
server_key: "2318f0d4eb928895446d5bd7222a0548"
server_key_iv: "b369e0eee1152cadfa386ed63d76d290"
server_key_version: "encryptionkey@0"
I20260925 10:08:26.289755 20147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:26.297703 20147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:26.300211 20147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:26.300377 20147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:26.300622 20147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:26.300937 20147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:26.300992 20147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:26.301036 20147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:26.301079 20147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:26.307384 20147 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:34277
I20260925 10:08:26.307439 20275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:34277 every 8 connection(s)
I20260925 10:08:26.307758 20147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:08:26.311740 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 20147
I20260925 10:08:26.311933 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:08:26.312144 14501 external_mini_cluster.cc:1428] Setting key 0932dafec1b8a2bf6e4771fd08002f62
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:26.318742 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:26.308526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:58087 (local address 127.14.41.126:40387)
0925 10:08:26.308591 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:26.308593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:26.308806 (+ 213us) server_negotiation.cc:407] Connection header received
0925 10:08:26.309849 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:26.309850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:26.309866 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:26.309936 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:26.310649 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:26.311243 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:26.312760 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:26.313023 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:26.313102 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:26.315654 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:26.315656 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:26.315656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:26.315664 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:26.317264 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:26.317699 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:26.317700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:26.317701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:26.317766 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:26.318044 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:26.318045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:26.318046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:26.318171 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:26.318254 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:26.318558 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 10:08:26.318650 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:08:26.318957 20278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:26.308486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:40387 (local address 127.14.41.65:58087)
0925 10:08:26.308665 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:26.308708 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:26.309650 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:26.310020 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:26.310023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:26.310286 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:26.310545 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:26.310549 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:26.311371 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:26.311372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:26.312638 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:26.312640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:26.313114 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:26.313115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:26.313263 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:26.313852 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:26.313861 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:26.315539 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:26.317360 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:26.317369 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:26.317371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:26.317617 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:26.317836 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:26.317837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:26.317837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:26.317968 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:26.318255 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:26.318264 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:26.318415 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 10:08:26.318524 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 10:08:26.318692 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":106,"threads_started":1}
I20260925 10:08:26.319514 20276 heartbeater.cc:347] Connected to a master server at 127.14.41.126:40387
I20260925 10:08:26.319614 20276 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:26.319825 20276 heartbeater.cc:511] Master 127.14.41.126:40387 requested a full tablet report, sending...
I20260925 10:08:26.320482 20085 ts_manager.cc:194] Registered new tserver with Master: 60518bf6cbed481688fff5a5c235155f (127.14.41.65:34277)
WARNING: no policy specified for kudu/127.14.41.66@KRBTEST.COM; defaulting to no policy
I20260925 10:08:26.321305 20085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:58087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
I20260925 10:08:26.340373 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:40387
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:26.424315 20281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:26.424571 20281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:26.424607 20281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:26.425792 20281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:26.425863 20281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:26.425906 20281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:08:26.427531 20281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:40387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:26.427812 20281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:26.428086 20281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:26.433614 20286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:26.433634 20289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:26.433634 20287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:26.433851 20281 server_base.cc:1034] running on GCE node
I20260925 10:08:26.434149 20281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:26.434407 20281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:26.435580 20281 hybrid_clock.cc:648] HybridClock initialized: now 1790330906435561 us; error 57 us; skew 500 ppm
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:26.438211 20281 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:08:26.438732 20281 webserver.cc:460] Webserver started at http://127.14.41.66:44269/ using document root <none> and password file <none>
I20260925 10:08:26.438969 20281 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:26.439041 20281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:26.439195 20281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:26.440328 20281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "4f6e6aad45a8448b8d3a83b3bec147c5"
format_stamp: "Formatted at 2026-09-25 10:08:26 on dist-test-slave-s6tm"
server_key: "7b3d8b4dff4478f7ac81bee8ae7513b5"
server_key_iv: "59c023f32c431e27318529ffb0906ec6"
server_key_version: "encryptionkey@0"
I20260925 10:08:26.440711 20281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "4f6e6aad45a8448b8d3a83b3bec147c5"
format_stamp: "Formatted at 2026-09-25 10:08:26 on dist-test-slave-s6tm"
server_key: "7b3d8b4dff4478f7ac81bee8ae7513b5"
server_key_iv: "59c023f32c431e27318529ffb0906ec6"
server_key_version: "encryptionkey@0"
I20260925 10:08:26.442108 20281 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:08:26.442912 20296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:26.443121 20281 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:26.443229 20281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "4f6e6aad45a8448b8d3a83b3bec147c5"
format_stamp: "Formatted at 2026-09-25 10:08:26 on dist-test-slave-s6tm"
server_key: "7b3d8b4dff4478f7ac81bee8ae7513b5"
server_key_iv: "59c023f32c431e27318529ffb0906ec6"
server_key_version: "encryptionkey@0"
I20260925 10:08:26.443307 20281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:26.466904 20281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:26.469532 20281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:26.469726 20281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:26.470065 20281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:26.470427 20281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:26.470485 20281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:26.470521 20281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:26.470549 20281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:26.476831 20281 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:40253
I20260925 10:08:26.476974 20409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:40253 every 8 connection(s)
I20260925 10:08:26.477272 20281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:26.485958 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 20281
I20260925 10:08:26.486053 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:08:26.486366 14501 external_mini_cluster.cc:1428] Setting key 5117a167d56e52dd86ab94c2845f399f
I20260925 10:08:26.488219 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:26.478114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:49631 (local address 127.14.41.126:40387)
0925 10:08:26.478202 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:26.478204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:26.478387 (+ 183us) server_negotiation.cc:407] Connection header received
0925 10:08:26.479406 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:26.479406 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:26.479416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:26.479485 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:26.480263 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:26.480767 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:26.481807 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:26.482031 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:26.482088 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:26.484570 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:26.484571 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:26.484571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:26.484579 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:26.486216 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:26.486808 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:26.486810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:26.486810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:26.486958 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:26.487317 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:26.487318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:26.487318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:26.487468 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:26.487537 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:26.487972 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 10:08:26.488074 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:08:26.488320 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:26.478087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:40387 (local address 127.14.41.66:49631)
0925 10:08:26.478248 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:26.478298 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:26.479302 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:26.479546 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:26.479549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:26.479965 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:26.480186 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:26.480191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:26.480893 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:26.480894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:26.481716 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:26.481718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:26.482112 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:26.482113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:26.482221 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:26.482879 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:26.482888 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:26.484478 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:26.486352 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:26.486363 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:26.486365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:26.486671 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:26.487087 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:26.487089 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:26.487089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:26.487252 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:26.487541 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:26.487547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:26.487683 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 10:08:26.487898 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 10:08:26.488103 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260925 10:08:26.488878 20410 heartbeater.cc:347] Connected to a master server at 127.14.41.126:40387
I20260925 10:08:26.489001 20410 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:26.489207 20410 heartbeater.cc:511] Master 127.14.41.126:40387 requested a full tablet report, sending...
I20260925 10:08:26.489639 20085 ts_manager.cc:194] Registered new tserver with Master: 4f6e6aad45a8448b8d3a83b3bec147c5 (127.14.41.66:40253)
I20260925 10:08:26.490058 20085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:49631
WARNING: no policy specified for kudu/127.14.41.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
I20260925 10:08:26.515396 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:40387
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:26.599355 20415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:26.599618 20415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:26.599663 20415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:26.600968 20415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:26.601044 20415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:26.601092 20415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:08:26.602651 20415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:40387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:26.602944 20415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:26.603195 20415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:26.608788 20420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:26.608825 20415 server_base.cc:1034] running on GCE node
W20260925 10:08:26.608875 20421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:26.609015 20423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:26.609254 20415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:26.609536 20415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:26.610715 20415 hybrid_clock.cc:648] HybridClock initialized: now 1790330906610669 us; error 78 us; skew 500 ppm
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:26.613675 20415 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:08:26.614151 20415 webserver.cc:460] Webserver started at http://127.14.41.67:34371/ using document root <none> and password file <none>
I20260925 10:08:26.614414 20415 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:26.614521 20415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:26.614673 20415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:26.615747 20415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "f6fd1da80bd74d178283cf2b7e70ca9b"
format_stamp: "Formatted at 2026-09-25 10:08:26 on dist-test-slave-s6tm"
server_key: "fc7096fdb17c1bf351fb36cb1acfb7df"
server_key_iv: "07c8527b8acc1bdd267bf73630fd1834"
server_key_version: "encryptionkey@0"
I20260925 10:08:26.616147 20415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "f6fd1da80bd74d178283cf2b7e70ca9b"
format_stamp: "Formatted at 2026-09-25 10:08:26 on dist-test-slave-s6tm"
server_key: "fc7096fdb17c1bf351fb36cb1acfb7df"
server_key_iv: "07c8527b8acc1bdd267bf73630fd1834"
server_key_version: "encryptionkey@0"
I20260925 10:08:26.617570 20415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:26.618542 20430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:26.618817 20415 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:08:26.618942 20415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "f6fd1da80bd74d178283cf2b7e70ca9b"
format_stamp: "Formatted at 2026-09-25 10:08:26 on dist-test-slave-s6tm"
server_key: "fc7096fdb17c1bf351fb36cb1acfb7df"
server_key_iv: "07c8527b8acc1bdd267bf73630fd1834"
server_key_version: "encryptionkey@0"
I20260925 10:08:26.619037 20415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:26.643649 20415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:26.646103 20415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:26.646371 20415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:26.646646 20415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:26.646988 20415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:26.647045 20415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:26.647081 20415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:26.647112 20415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:26.653795 20415 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:34713
I20260925 10:08:26.653846 20543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:34713 every 8 connection(s)
I20260925 10:08:26.654199 20415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:08:26.660988 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 20415
I20260925 10:08:26.661100 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330906, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:26.661302 14501 external_mini_cluster.cc:1428] Setting key d65abcd79b5631d97bd11ce130e59df5
I20260925 10:08:26.664973 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:26.655016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:45505 (local address 127.14.41.126:40387)
0925 10:08:26.655103 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:26.655105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:26.655200 (+ 95us) server_negotiation.cc:407] Connection header received
0925 10:08:26.656310 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:26.656311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:26.656323 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:26.656402 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:26.657185 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:26.657686 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:26.658975 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:26.659168 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:26.659230 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:26.661797 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:26.661799 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:26.661799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:26.661814 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:26.663430 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:26.664031 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:26.664032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:26.664033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:26.664112 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:26.664363 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:26.664364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:26.664364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:26.664489 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:26.664548 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:26.664827 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 10:08:26.664905 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:08:26.665122 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:26.654928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:40387 (local address 127.14.41.67:45505)
0925 10:08:26.655087 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:26.655128 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:26.656181 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:26.656484 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:26.656487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:26.656868 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:26.657088 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:26.657093 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:26.657954 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:26.657956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:26.658868 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:26.658871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:26.659279 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:26.659281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:26.659438 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:26.660134 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:26.660144 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:26.661683 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:26.663516 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:26.663523 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:26.663525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:26.663761 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:26.664184 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:26.664184 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:26.664185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:26.664297 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:26.664551 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:26.664556 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:26.664684 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 10:08:26.664772 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 10:08:26.664904 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260925 10:08:26.665678 20544 heartbeater.cc:347] Connected to a master server at 127.14.41.126:40387
I20260925 10:08:26.665782 20544 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:26.666026 20544 heartbeater.cc:511] Master 127.14.41.126:40387 requested a full tablet report, sending...
I20260925 10:08:26.666579 20085 ts_manager.cc:194] Registered new tserver with Master: f6fd1da80bd74d178283cf2b7e70ca9b (127.14.41.67:34713)
I20260925 10:08:26.667142 20085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:45505
I20260925 10:08:26.673159 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/test-admin.keytab.
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:26.690280 14501 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 10:08:26.692224 20551 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:26.691368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43352 (local address 127.14.41.65:34277)
0925 10:08:26.691547 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:26.691552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:26.691562 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:08:26.691588 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:26.691589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:26.691604 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:26.691693 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:26.692124 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43352: BlockingRecv error: recv got EOF from 127.0.0.1:43352 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
W20260925 10:08:26.692433 14501 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/krb5cc' of type FILE
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:26.693754 14501 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:26.694832 14501 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.65@KRBTEST.COM
I20260925 10:08:26.704557 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:26.695409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43354 (local address 127.14.41.65:34277)
0925 10:08:26.695476 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:26.695478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:26.695536 (+ 58us) server_negotiation.cc:407] Connection header received
0925 10:08:26.695587 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:26.695588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:26.695600 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:26.695656 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:26.696176 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:26.697254 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:26.698032 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:26.698424 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:26.698496 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:26.701080 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:26.701089 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:26.701089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:26.701096 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:26.703046 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:26.703517 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:26.703521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:26.703521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:26.703589 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:26.703800 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:26.703801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:26.703801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:26.704182 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:26.704320 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:26.704407 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 10:08:26.704475 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 10:08:26.706871 20551 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:26.706270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43368 (local address 127.14.41.65:34277)
0925 10:08:26.706311 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:26.706313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:26.706337 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:08:26.706398 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:26.706399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:26.706411 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:26.706482 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:26.706801 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43368: BlockingRecv error: recv got EOF from 127.0.0.1:43368 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 10:08:26.707188 14501 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/krb5cc)
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:26.708746 14501 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:26.709833 14501 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.65@KRBTEST.COM
I20260925 10:08:26.718447 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:26.710598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43372 (local address 127.14.41.65:34277)
0925 10:08:26.710673 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:26.710676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:26.710771 (+ 95us) server_negotiation.cc:407] Connection header received
0925 10:08:26.710951 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:26.710952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:26.710972 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:26.711038 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:26.711493 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:26.712098 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:26.712923 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:26.713233 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:26.713338 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:26.716015 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:26.716016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:26.716017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:26.716024 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:26.717407 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:26.717761 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:26.717761 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:26.717762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:26.717818 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:26.718006 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:26.718006 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:26.718006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:26.718105 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:26.718184 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:26.718305 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 10:08:26.718372 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 10:08:26.720299 20551 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:26.719556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.14.41.65:34277)
0925 10:08:26.719619 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:26.719621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:26.719693 (+ 72us) server_negotiation.cc:407] Connection header received
0925 10:08:26.719772 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:26.719773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:26.719784 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:26.719837 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:26.720242 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43374: BlockingRecv error: recv got EOF from 127.0.0.1:43374 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 10:08:26.720475 14501 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790330877480195-14501-0/krb5kdc/krb5cc' of type FILE
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:26.721688 14501 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:26.722891 14501 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.65@KRBTEST.COM
I20260925 10:08:26.731464 20551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:26.723559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43386 (local address 127.14.41.65:34277)
0925 10:08:26.723628 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:26.723631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:26.723705 (+ 74us) server_negotiation.cc:407] Connection header received
0925 10:08:26.723810 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:26.723811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:26.723822 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:26.723901 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:26.724341 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:26.724880 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:26.725719 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:26.725941 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:26.726039 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:26.728644 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:26.728645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:26.728645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:26.728654 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:26.730240 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:26.730633 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:26.730634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:26.730635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:26.730699 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:26.730925 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:26.730926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:26.730927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:26.731039 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:26.731150 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:26.731268 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 10:08:26.731351 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:08:26.732280 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 20147
I20260925 10:08:26.737143 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 20281
I20260925 10:08:26.741600 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 20415
I20260925 10:08:26.746292 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 20053
2026-09-25T10:08:26Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1949 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20564](info): set up 2 sockets
Sep 25 10:08:26 dist-test-slave-s6tm krb5kdc[20564](info): commencing operation
krb5kdc: starting...
W20260925 10:08:27.798020 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:08:27 dist-test-slave-s6tm krb5kdc[20564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:08:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.126.keytab.
Entry for principal oryx/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.126.keytab.
I20260925 10:08:27.922602 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:43871
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.126.keytab
--principal=oryx/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:43871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:28.008208 20577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:28.008435 20577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:28.008459 20577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:28.009725 20577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:28.009775 20577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:28.009790 20577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:28.009804 20577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:28.009815 20577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:08:28.011411 20577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:43871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.126.keytab
--principal=oryx/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:43871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:28.011695 20577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:28.011983 20577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:28.017100 20585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:28.017108 20582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:28.017122 20583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:28.017469 20577 server_base.cc:1034] running on GCE node
I20260925 10:08:28.017750 20577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:28.018054 20577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:28.019210 20577 hybrid_clock.cc:648] HybridClock initialized: now 1790330908019186 us; error 39 us; skew 500 ppm
Sep 25 10:08:28 dist-test-slave-s6tm krb5kdc[20564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330908, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:28.025345 20577 init.cc:377] Logged in from keytab as oryx/127.14.41.126@KRBTEST.COM (short username oryx)
I20260925 10:08:28.025770 20577 webserver.cc:460] Webserver started at http://127.14.41.126:42205/ using document root <none> and password file <none>
I20260925 10:08:28.025961 20577 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:28.026003 20577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:28.026095 20577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:28.027204 20577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "efc62ed2149648ee810e263591ecca7d"
format_stamp: "Formatted at 2026-09-25 10:08:28 on dist-test-slave-s6tm"
server_key: "5710aac63b6780873af48d352c8de6d8"
server_key_iv: "1c6584f791389c29309c8a9852b6896f"
server_key_version: "encryptionkey@0"
I20260925 10:08:28.027550 20577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "efc62ed2149648ee810e263591ecca7d"
format_stamp: "Formatted at 2026-09-25 10:08:28 on dist-test-slave-s6tm"
server_key: "5710aac63b6780873af48d352c8de6d8"
server_key_iv: "1c6584f791389c29309c8a9852b6896f"
server_key_version: "encryptionkey@0"
I20260925 10:08:28.028956 20577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:08:28.029767 20592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:28.030022 20577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:28.030133 20577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "efc62ed2149648ee810e263591ecca7d"
format_stamp: "Formatted at 2026-09-25 10:08:28 on dist-test-slave-s6tm"
server_key: "5710aac63b6780873af48d352c8de6d8"
server_key_iv: "1c6584f791389c29309c8a9852b6896f"
server_key_version: "encryptionkey@0"
I20260925 10:08:28.030212 20577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:28.038616 20577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:28.041250 20577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:28.041435 20577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:28.045629 20577 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:43871
I20260925 10:08:28.045666 20644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:43871 every 8 connection(s)
I20260925 10:08:28.046056 20577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:28.047197 20645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:28.048054 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 20577
I20260925 10:08:28.048182 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:08:28.048412 14501 external_mini_cluster.cc:1428] Setting key 7d3a80ec114daaad10dea71f06a7ccf2
I20260925 10:08:28.051002 20645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d: Bootstrap starting.
I20260925 10:08:28.051712 20645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:28.052136 20645 log.cc:824] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:28.053017 20645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d: No bootstrap required, opened a new log
I20260925 10:08:28.055997 20645 raft_consensus.cc:348] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } }
I20260925 10:08:28.056170 20645 raft_consensus.cc:374] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:28.056197 20645 raft_consensus.cc:729] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efc62ed2149648ee810e263591ecca7d, State: Initialized, Role: FOLLOWER
I20260925 10:08:28.056365 20645 consensus_queue.cc:260] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } }
I20260925 10:08:28.056463 20645 raft_consensus.cc:388] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 25 10:08:28 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM
I20260925 10:08:28.056527 20645 raft_consensus.cc:482] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:28.056579 20645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:28.057214 20645 raft_consensus.cc:504] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } }
I20260925 10:08:28.057535 20645 leader_election.cc:302] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efc62ed2149648ee810e263591ecca7d; no voters:
I20260925 10:08:28.057771 20645 leader_election.cc:288] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:08:28.057804 20650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:28.057910 20650 raft_consensus.cc:686] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [term 1 LEADER]: Becoming Leader. State: Replica: efc62ed2149648ee810e263591ecca7d, State: Running, Role: LEADER
I20260925 10:08:28.057997 20650 consensus_queue.cc:237] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } }
I20260925 10:08:28.058099 20645 sys_catalog.cc:564] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:28.058303 20650 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efc62ed2149648ee810e263591ecca7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } }
I20260925 10:08:28.058372 20650 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:28.058507 20650 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader efc62ed2149648ee810e263591ecca7d. Latest consensus state: current_term: 1 leader_uuid: "efc62ed2149648ee810e263591ecca7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } }
I20260925 10:08:28.058598 20654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:28.058665 20650 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:28.059216 20654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:28.061115 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.049025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38058 (local address 127.14.41.126:43871)
0925 10:08:28.049202 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:28.049218 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:28.049247 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:08:28.049679 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:28.049687 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:28.049738 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:28.049963 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:28.050909 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.051993 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.053155 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.053634 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.053832 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.057048 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:28.057053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:28.057053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:28.057067 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:28.059273 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.059697 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.059700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.059701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.059811 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.060079 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.060080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.060080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.060329 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:28.060517 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:28.060710 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 10:08:28.060858 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260925 10:08:28.064575 20654 catalog_manager.cc:1347] Generated new cluster ID: 5b434835f51047d689e6ab46671765f6
I20260925 10:08:28.064661 20654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:08:28.080525 20654 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:08:28.081388 20654 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:28.087836 20654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d: Generated new TSK 0
I20260925 10:08:28.088120 20654 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.65.keytab.
Entry for principal oryx/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.65.keytab.
I20260925 10:08:28.124043 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:43871
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.65.keytab
--principal=oryx/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:28.229010 20671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:28.229230 20671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:28.229291 20671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:28.230448 20671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:28.230494 20671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:28.230521 20671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:08:28.232012 20671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.65.keytab
--principal=oryx/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:43871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:28.232229 20671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:28.232409 20671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:08:28.237648 20671 server_base.cc:1034] running on GCE node
W20260925 10:08:28.237679 20677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:28.237679 20676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:28.237761 20679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:28.238114 20671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:28.238394 20671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:28.239559 20671 hybrid_clock.cc:648] HybridClock initialized: now 1790330908239549 us; error 37 us; skew 500 ppm
Sep 25 10:08:28 dist-test-slave-s6tm krb5kdc[20564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330908, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:28.242422 20671 init.cc:377] Logged in from keytab as oryx/127.14.41.65@KRBTEST.COM (short username oryx)
I20260925 10:08:28.242879 20671 webserver.cc:460] Webserver started at http://127.14.41.65:35207/ using document root <none> and password file <none>
I20260925 10:08:28.243127 20671 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:28.243196 20671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:28.243355 20671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:28.244496 20671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "e02b6aac6c8f403396d2a6cfe768d541"
format_stamp: "Formatted at 2026-09-25 10:08:28 on dist-test-slave-s6tm"
server_key: "9bd8d8ec78d0bc120419b51d2fc224f1"
server_key_iv: "3cbef466bfc27301059d1e4ebb402c93"
server_key_version: "encryptionkey@0"
I20260925 10:08:28.244988 20671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "e02b6aac6c8f403396d2a6cfe768d541"
format_stamp: "Formatted at 2026-09-25 10:08:28 on dist-test-slave-s6tm"
server_key: "9bd8d8ec78d0bc120419b51d2fc224f1"
server_key_iv: "3cbef466bfc27301059d1e4ebb402c93"
server_key_version: "encryptionkey@0"
I20260925 10:08:28.246333 20671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:08:28.247159 20686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:28.247370 20671 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:28.247476 20671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "e02b6aac6c8f403396d2a6cfe768d541"
format_stamp: "Formatted at 2026-09-25 10:08:28 on dist-test-slave-s6tm"
server_key: "9bd8d8ec78d0bc120419b51d2fc224f1"
server_key_iv: "3cbef466bfc27301059d1e4ebb402c93"
server_key_version: "encryptionkey@0"
I20260925 10:08:28.247552 20671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:28.259229 20671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:28.261936 20671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:28.262111 20671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:28.262356 20671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:28.262694 20671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:28.262756 20671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:28.262794 20671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:28.262821 20671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:28.268822 20671 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:40287
I20260925 10:08:28.268935 20799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:40287 every 8 connection(s)
I20260925 10:08:28.269263 20671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:08:28.270109 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 20671
I20260925 10:08:28.270213 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:08:28.270469 14501 external_mini_cluster.cc:1428] Setting key b1f2f2c652fa96382e339f3705e80edb
Sep 25 10:08:28 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330908, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.14.41.66@KRBTEST.COM; defaulting to no policy
I20260925 10:08:28.280088 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.270124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:32935 (local address 127.14.41.126:43871)
0925 10:08:28.270178 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:28.270180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:28.270336 (+ 156us) server_negotiation.cc:407] Connection header received
0925 10:08:28.271061 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:28.271063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:28.271074 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:28.271115 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:28.272578 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.273206 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.274330 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.274558 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.274652 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.277009 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:28.277011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:28.277012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:28.277020 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:28.278514 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.278969 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.278970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.278971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.279031 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.279318 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.279319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.279320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.279454 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:28.279526 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:28.279874 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 10:08:28.279991 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 10:08:28.280308 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.269988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.65:32935)
0925 10:08:28.270184 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:28.270241 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:28.270956 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:28.271201 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:28.271205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:28.272137 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:28.272436 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:28.272443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.273366 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.273368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:28.274217 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:28.274219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.274665 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.274666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:28.274837 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.275440 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:28.275448 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:28.276897 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:28.278619 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:28.278628 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:28.278630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:28.278872 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:28.279101 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:28.279102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:28.279103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:28.279237 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:28.279529 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:28.279536 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:28.279727 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 10:08:28.279842 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 10:08:28.280040 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":108,"threads_started":1}
I20260925 10:08:28.280900 20800 heartbeater.cc:347] Connected to a master server at 127.14.41.126:43871
I20260925 10:08:28.280989 20800 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:28.281173 20800 heartbeater.cc:511] Master 127.14.41.126:43871 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.41.66@KRBTEST.COM" created.
I20260925 10:08:28.281711 20609 ts_manager.cc:194] Registered new tserver with Master: e02b6aac6c8f403396d2a6cfe768d541 (127.14.41.65:40287)
I20260925 10:08:28.282328 20609 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:32935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.66.keytab.
Entry for principal oryx/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.66.keytab.
I20260925 10:08:28.300671 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:43871
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.66.keytab
--principal=oryx/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:28.385406 20805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:28.385708 20805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:28.385733 20805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:28.386889 20805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:28.386940 20805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:28.386967 20805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:08:28.388409 20805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.66.keytab
--principal=oryx/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:43871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:28.388602 20805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:28.388788 20805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:28.394223 20811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:28.394295 20813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:28.394220 20810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:28.394981 20805 server_base.cc:1034] running on GCE node
I20260925 10:08:28.395226 20805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:28.395467 20805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:28.396631 20805 hybrid_clock.cc:648] HybridClock initialized: now 1790330908396609 us; error 42 us; skew 500 ppm
Sep 25 10:08:28 dist-test-slave-s6tm krb5kdc[20564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330908, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:28.399662 20805 init.cc:377] Logged in from keytab as oryx/127.14.41.66@KRBTEST.COM (short username oryx)
I20260925 10:08:28.400208 20805 webserver.cc:460] Webserver started at http://127.14.41.66:46717/ using document root <none> and password file <none>
I20260925 10:08:28.400427 20805 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:28.400472 20805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:28.400573 20805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:28.401721 20805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "949ae6782e9d46b59871d4d1f6a05607"
format_stamp: "Formatted at 2026-09-25 10:08:28 on dist-test-slave-s6tm"
server_key: "03ff6fa824d81ffaf8ca3b4e3e722a1e"
server_key_iv: "e823724dc754bec211f2d528a2be206d"
server_key_version: "encryptionkey@0"
I20260925 10:08:28.402086 20805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "949ae6782e9d46b59871d4d1f6a05607"
format_stamp: "Formatted at 2026-09-25 10:08:28 on dist-test-slave-s6tm"
server_key: "03ff6fa824d81ffaf8ca3b4e3e722a1e"
server_key_iv: "e823724dc754bec211f2d528a2be206d"
server_key_version: "encryptionkey@0"
I20260925 10:08:28.403576 20805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:08:28.404618 20820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:28.404819 20805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:28.404937 20805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "949ae6782e9d46b59871d4d1f6a05607"
format_stamp: "Formatted at 2026-09-25 10:08:28 on dist-test-slave-s6tm"
server_key: "03ff6fa824d81ffaf8ca3b4e3e722a1e"
server_key_iv: "e823724dc754bec211f2d528a2be206d"
server_key_version: "encryptionkey@0"
I20260925 10:08:28.405004 20805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:28.422711 20805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:28.425089 20805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:28.425213 20805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:28.425480 20805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:28.425761 20805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:28.425794 20805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:28.425817 20805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:28.425830 20805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:28.431782 20805 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:44663
I20260925 10:08:28.431841 20933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:44663 every 8 connection(s)
I20260925 10:08:28.432186 20805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:08:28.436441 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 20805
I20260925 10:08:28.436544 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:08:28.436885 14501 external_mini_cluster.cc:1428] Setting key 29d545820ef235d0d2e0116414580034
Sep 25 10:08:28 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330908, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM
I20260925 10:08:28.443879 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.433248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:54905 (local address 127.14.41.126:43871)
0925 10:08:28.433323 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:28.433327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:28.433579 (+ 252us) server_negotiation.cc:407] Connection header received
0925 10:08:28.434240 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:28.434242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:28.434255 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:28.434328 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:28.435359 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.436112 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.437461 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.437746 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.437845 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.440403 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:28.440405 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:28.440405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:28.440414 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:28.442187 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.442720 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.442722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.442722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.442794 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.443092 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.443093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.443093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.443237 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:28.443362 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:28.443682 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 10:08:28.443781 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:08:28.444250 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.433233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.66:54905)
0925 10:08:28.433447 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:28.433486 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:28.434105 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:28.434456 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:28.434460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:28.435039 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:28.435238 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:28.435242 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.436200 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.436201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:28.437374 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:28.437377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.437846 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.437847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:28.437981 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.438566 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:28.438575 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:28.440297 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:28.442302 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:28.442310 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:28.442312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:28.442582 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:28.442872 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:28.442873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:28.442874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:28.443012 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:28.443367 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:28.443375 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:28.443535 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 10:08:28.443655 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 10:08:28.443992 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":114,"threads_started":1}
I20260925 10:08:28.444753 20934 heartbeater.cc:347] Connected to a master server at 127.14.41.126:43871
I20260925 10:08:28.444891 20934 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:28.445122 20934 heartbeater.cc:511] Master 127.14.41.126:43871 requested a full tablet report, sending...
I20260925 10:08:28.445647 20609 ts_manager.cc:194] Registered new tserver with Master: 949ae6782e9d46b59871d4d1f6a05607 (127.14.41.66:44663)
WARNING: no policy specified for oryx/127.14.41.67@KRBTEST.COM; defaulting to no policy
I20260925 10:08:28.446144 20609 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:54905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.41.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.67.keytab.
Entry for principal oryx/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.67.keytab.
I20260925 10:08:28.469050 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:43871
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.67.keytab
--principal=oryx/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:28.558779 20939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:28.559027 20939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:28.559108 20939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:28.560470 20939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:28.560520 20939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:28.560549 20939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:08:28.561990 20939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.67.keytab
--principal=oryx/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:43871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:28.562202 20939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:28.562387 20939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:28.567662 20945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:28.567656 20944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:28.567656 20947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:28.567821 20939 server_base.cc:1034] running on GCE node
I20260925 10:08:28.568181 20939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:28.568413 20939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:28.569594 20939 hybrid_clock.cc:648] HybridClock initialized: now 1790330908569583 us; error 41 us; skew 500 ppm
Sep 25 10:08:28 dist-test-slave-s6tm krb5kdc[20564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330908, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:28.572602 20939 init.cc:377] Logged in from keytab as oryx/127.14.41.67@KRBTEST.COM (short username oryx)
I20260925 10:08:28.573168 20939 webserver.cc:460] Webserver started at http://127.14.41.67:46291/ using document root <none> and password file <none>
I20260925 10:08:28.573427 20939 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:28.573498 20939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:28.573652 20939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:28.574797 20939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "bb35bd8e8a9f42b1aa31926ca80f711b"
format_stamp: "Formatted at 2026-09-25 10:08:28 on dist-test-slave-s6tm"
server_key: "2e21832034ebc12b2825cecb739a8ffb"
server_key_iv: "bb4ed37b5f3e8aae2d042c342acccde2"
server_key_version: "encryptionkey@0"
I20260925 10:08:28.575184 20939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "bb35bd8e8a9f42b1aa31926ca80f711b"
format_stamp: "Formatted at 2026-09-25 10:08:28 on dist-test-slave-s6tm"
server_key: "2e21832034ebc12b2825cecb739a8ffb"
server_key_iv: "bb4ed37b5f3e8aae2d042c342acccde2"
server_key_version: "encryptionkey@0"
I20260925 10:08:28.576679 20939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:08:28.577605 20954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:28.577934 20939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:28.578086 20939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "bb35bd8e8a9f42b1aa31926ca80f711b"
format_stamp: "Formatted at 2026-09-25 10:08:28 on dist-test-slave-s6tm"
server_key: "2e21832034ebc12b2825cecb739a8ffb"
server_key_iv: "bb4ed37b5f3e8aae2d042c342acccde2"
server_key_version: "encryptionkey@0"
I20260925 10:08:28.578177 20939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:28.588449 20939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:28.591092 20939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:28.591284 20939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:28.591571 20939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:28.591902 20939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:28.592038 20939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:28.592087 20939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:28.592118 20939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:28.598196 20939 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:44021
I20260925 10:08:28.598233 21067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:44021 every 8 connection(s)
I20260925 10:08:28.598592 20939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
Sep 25 10:08:28 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330908, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM
I20260925 10:08:28.605032 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 20939
I20260925 10:08:28.605144 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:08:28.605345 14501 external_mini_cluster.cc:1428] Setting key 040ba90a1ec1eb01020fe4e159b0a5d1
I20260925 10:08:28.608533 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.599358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:32827 (local address 127.14.41.126:43871)
0925 10:08:28.599435 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:28.599438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:28.599655 (+ 217us) server_negotiation.cc:407] Connection header received
0925 10:08:28.600323 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:28.600324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:28.600335 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:28.600391 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:28.601393 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.601907 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.602935 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.603124 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.603176 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.605666 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:28.605669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:28.605670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:28.605679 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:28.607177 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.607597 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.607598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.607599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.607651 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.607890 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.607890 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.607891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.608098 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:28.608150 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:28.608398 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 10:08:28.608467 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 10:08:28.608777 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.599338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.67:32827)
0925 10:08:28.599534 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:28.599571 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:28.600229 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:28.600478 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:28.600482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:28.601096 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:28.601299 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:28.601304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.602005 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.602006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:28.602858 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:28.602861 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.603196 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.603197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:28.603297 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.603788 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:28.603796 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:28.605519 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:28.607271 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:28.607278 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:28.607280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:28.607525 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:28.607713 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:28.607713 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:28.607714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:28.607825 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:28.608155 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:28.608160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:28.608284 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 10:08:28.608371 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 10:08:28.608505 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":110,"threads_started":1}
I20260925 10:08:28.609486 21068 heartbeater.cc:347] Connected to a master server at 127.14.41.126:43871
I20260925 10:08:28.609609 21068 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:28.609794 21068 heartbeater.cc:511] Master 127.14.41.126:43871 requested a full tablet report, sending...
I20260925 10:08:28.610267 20609 ts_manager.cc:194] Registered new tserver with Master: bb35bd8e8a9f42b1aa31926ca80f711b (127.14.41.67:44021)
I20260925 10:08:28.610651 20609 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:32827
I20260925 10:08:28.617496 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.41.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.41.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.41.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.125.keytab.
Entry for principal oryx/127.14.41.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.125.keytab.
I20260925 10:08:28.646552 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.125:41713
--webserver_interface=127.14.41.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.125.keytab
--principal=oryx/127.14.41.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:43871,127.14.41.125:41713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:28.729831 21073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:28.730051 21073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:28.730073 21073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:28.731175 21073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:28.731244 21073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:28.731268 21073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:28.731288 21073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:28.731314 21073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:08:28.732993 21073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:43871,127.14.41.125:41713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.125.keytab
--principal=oryx/127.14.41.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.125:41713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.41.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
W20260925 10:08:28.733104 21073 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.14.41.126:43871,127.14.41.125:41713'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 10:08:28.733220 21073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:28.733487 21073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:08:28.738655 21073 server_base.cc:1034] running on GCE node
W20260925 10:08:28.738685 21081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:28.738833 21079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:28.738685 21078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:28.739140 21073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:28.739459 21073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:28.740604 21073 hybrid_clock.cc:648] HybridClock initialized: now 1790330908740586 us; error 35 us; skew 500 ppm
Sep 25 10:08:28 dist-test-slave-s6tm krb5kdc[20564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330908, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:28.743302 21073 init.cc:377] Logged in from keytab as oryx/127.14.41.125@KRBTEST.COM (short username oryx)
I20260925 10:08:28.743814 21073 webserver.cc:460] Webserver started at http://127.14.41.125:40195/ using document root <none> and password file <none>
I20260925 10:08:28.744113 21073 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:28.744189 21073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:28.744393 21073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:28.745522 21073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/data/instance:
uuid: "86de7b8cfd2f42429708bdc3fa39e704"
format_stamp: "Formatted at 2026-09-25 10:08:28 on dist-test-slave-s6tm"
server_key: "d8a6b70375099a6f959a1b8430916160"
server_key_iv: "0302daaeea471a10c29e9764eb46d0ab"
server_key_version: "encryptionkey@0"
I20260925 10:08:28.745913 21073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/wal/instance:
uuid: "86de7b8cfd2f42429708bdc3fa39e704"
format_stamp: "Formatted at 2026-09-25 10:08:28 on dist-test-slave-s6tm"
server_key: "d8a6b70375099a6f959a1b8430916160"
server_key_iv: "0302daaeea471a10c29e9764eb46d0ab"
server_key_version: "encryptionkey@0"
I20260925 10:08:28.747496 21073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:08:28.748409 21088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:28.748606 21073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:08:28.748708 21073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/wal
uuid: "86de7b8cfd2f42429708bdc3fa39e704"
format_stamp: "Formatted at 2026-09-25 10:08:28 on dist-test-slave-s6tm"
server_key: "d8a6b70375099a6f959a1b8430916160"
server_key_iv: "0302daaeea471a10c29e9764eb46d0ab"
server_key_version: "encryptionkey@0"
I20260925 10:08:28.748793 21073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:28.756443 21073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:28.759052 21073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:28.759238 21073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:28.763756 21073 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.125:41713
I20260925 10:08:28.763779 21140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.125:41713 every 8 connection(s)
I20260925 10:08:28.764252 21073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/data/info.pb
I20260925 10:08:28.765174 21141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:28.766767 21141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:08:28.772217 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 21073
Sep 25 10:08:28 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330908, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.125@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM
I20260925 10:08:28.777752 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.767546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38062 (local address 127.14.41.126:43871)
0925 10:08:28.767601 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:28.767603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:28.767786 (+ 183us) server_negotiation.cc:407] Connection header received
0925 10:08:28.768370 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:28.768372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:28.768384 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:28.768451 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:28.769307 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.769808 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.771143 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.771394 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.771464 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.774347 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:28.774349 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:28.774349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:28.774359 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:28.776150 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.776663 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.776665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.776665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.776723 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.777002 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.777003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.777004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.777160 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:28.777238 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:28.777564 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 10:08:28.777658 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:08:28.778031 21143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.767460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:38062)
0925 10:08:28.767644 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:28.767686 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:28.768243 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:28.768590 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:28.768595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:28.768981 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:28.769199 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:28.769204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.769920 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.769921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:28.771041 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:28.771044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.771496 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.771497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:28.771603 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.772274 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:28.772286 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:28.774257 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:28.776245 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:28.776252 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:28.776254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:28.776564 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:28.776801 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:28.776802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:28.776802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:28.776930 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:28.777243 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:28.777249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:28.777395 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 10:08:28.777540 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 10:08:28.777704 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":114,"threads_started":1}
Sep 25 10:08:28 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.41.125@KRBTEST.COM
I20260925 10:08:28.778752 21141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:08:28.782214 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.772826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59856 (local address 127.14.41.125:41713)
0925 10:08:28.772988 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:28.772992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:28.773094 (+ 102us) server_negotiation.cc:407] Connection header received
0925 10:08:28.773148 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:28.773150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:28.773169 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:28.773234 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:28.773989 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.774917 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.775890 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.776403 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.776465 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.779106 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:28.779112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:28.779112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:28.779118 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:28.780931 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.781354 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.781357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.781358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.781419 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.781647 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.781648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.781648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.781790 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:28.781974 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:28.782020 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 10:08:28.782095 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
Sep 25 10:08:28 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330908, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.125@KRBTEST.COM for oryx/127.14.41.125@KRBTEST.COM
I20260925 10:08:28.786526 21143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.779016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:41713 (local address 127.0.0.1:59864)
0925 10:08:28.779065 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:28.779077 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:28.779115 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:28.779386 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:28.779386 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:28.779544 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:28.779610 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:28.779613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.780616 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.780617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:28.781226 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:28.781229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.781646 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.781647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:28.781716 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.782194 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:28.782200 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:28.783482 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:28.785520 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:28.785521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:28.785522 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:28.785684 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:28.785891 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:28.785892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:28.785892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:28.785953 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:28.786195 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:28.786196 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:28.786232 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 10:08:28.786300 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 10:08:28.786396 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 10:08:28.786604 21146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.779065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59864 (local address 127.14.41.125:41713)
0925 10:08:28.779240 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:28.779242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:28.779249 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:08:28.779268 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:28.779268 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:28.779279 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:28.779346 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:28.779687 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.780521 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.781356 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.781572 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.781650 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.783565 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:28.783566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:28.783566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:28.783573 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:28.785391 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.785768 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.785769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.785770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.785826 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.786023 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.786024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.786024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.786129 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:28.786203 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:28.786327 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 10:08:28.786453 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":154,"thread_start_us":107,"threads_started":1}
I20260925 10:08:28.787873 21141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: Bootstrap starting.
I20260925 10:08:28.788578 21141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:28.788868 21141 log.cc:824] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:28.789620 21141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: No bootstrap required, opened a new log
I20260925 10:08:28.791100 21141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } }
I20260925 10:08:28.791227 21141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:28.791249 21141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86de7b8cfd2f42429708bdc3fa39e704, State: Initialized, Role: FOLLOWER
I20260925 10:08:28.791404 21141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } }
I20260925 10:08:28.791683 21148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } } }
I20260925 10:08:28.791729 21141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:28.791774 21148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:08:28.793671 21159 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:08:28.793737 21159 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:08:28.795254 21073 master_runner.cc:428] Detected that this master 86de7b8cfd2f42429708bdc3fa39e704 is joining an existing cluster
I20260925 10:08:28.795307 21073 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.41.125:41713
I20260925 10:08:28.803833 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.796468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38072 (local address 127.14.41.126:43871)
0925 10:08:28.796518 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:28.796520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:28.796698 (+ 178us) server_negotiation.cc:407] Connection header received
0925 10:08:28.796727 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:28.796728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:28.796739 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:28.796792 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:28.797220 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.797835 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.798878 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.799191 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.799270 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.800495 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:28.800497 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:28.800497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:28.800506 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:28.802520 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.802934 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.802935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.802935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.803003 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.803223 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.803224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.803225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.803367 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:28.803437 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:28.803644 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 10:08:28.803734 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:08:28.803866 21146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.796741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.14.41.125:41713)
0925 10:08:28.796776 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:28.796778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:28.796904 (+ 126us) server_negotiation.cc:407] Connection header received
0925 10:08:28.796998 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:28.796998 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:28.797007 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:28.797061 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:28.797408 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.798140 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.799077 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.799362 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.799428 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.800634 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:28.800636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:28.800636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:28.800643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:28.802397 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.802979 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.802980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.802980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.803026 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.803280 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.803280 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.803281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.803391 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:28.803432 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:28.803610 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 10:08:28.803761 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":21}
I20260925 10:08:28.803860 21166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.796626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:41713 (local address 127.0.0.1:59880)
0925 10:08:28.796733 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:28.796742 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:28.796920 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:28.797065 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:28.797066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:28.797269 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:28.797324 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:28.797326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.798228 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.798229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:28.799004 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:28.799005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.799451 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.799452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:28.799522 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.800072 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:28.800078 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:28.800548 (+ 470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:28.802490 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:28.802491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:28.802492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:28.802881 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:28.803093 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:28.803094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:28.803094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:28.803201 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:28.803473 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:28.803474 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:28.803507 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 10:08:28.803588 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 10:08:28.803710 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260925 10:08:28.804224 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.796396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:38072)
0925 10:08:28.796557 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:28.796569 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:28.796639 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:28.796844 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:28.796845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:28.797049 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:28.797135 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:28.797137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.797921 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.797923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:28.798785 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:28.798787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.799294 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.799295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:28.799359 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.799873 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:28.799878 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:28.800385 (+ 507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:28.802587 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:28.802588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:28.802588 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:28.802858 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:28.803070 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:28.803071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:28.803071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:28.803156 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:28.803449 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:28.803450 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:28.803508 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 10:08:28.804011 (+ 503us) client_negotiation.cc:240] Negotiation successful
0925 10:08:28.804115 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
Sep 25 10:08:28 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330908, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.126@KRBTEST.COM for oryx/127.14.41.125@KRBTEST.COM
I20260925 10:08:28.815001 21146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.806631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59886 (local address 127.14.41.125:41713)
0925 10:08:28.806682 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:28.806684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:28.806853 (+ 169us) server_negotiation.cc:407] Connection header received
0925 10:08:28.806940 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:28.806941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:28.806952 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:28.807009 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:28.807592 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.808231 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.809527 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.809705 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.809776 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.812167 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:28.812169 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:28.812169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:28.812178 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:28.813693 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.814131 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.814133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.814133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.814194 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.814410 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.814411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.814412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.814532 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:28.814609 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:28.814802 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 10:08:28.814895 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 10:08:28.815017 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.806544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:41713 (local address 127.0.0.1:59886)
0925 10:08:28.806726 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:28.806773 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:28.806853 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:28.807029 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:28.807033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:28.807385 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:28.807459 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:28.807461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.808338 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.808340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:28.809382 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:28.809385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.809784 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.809786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:28.809874 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.810445 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:28.810450 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:28.812072 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:28.813803 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:28.813812 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:28.813815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:28.814037 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:28.814266 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:28.814267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:28.814268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:28.814344 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:28.814601 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:28.814609 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:28.814669 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 10:08:28.814773 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 10:08:28.814906 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":114,"threads_started":1}
I20260925 10:08:28.815580 20609 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.41.125:41713: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704"
member_type: NON_VOTER
last_known_addr {
host: "127.14.41.125"
port: 41713
}
attrs {
promote: true
}
}
dest_uuid: "efc62ed2149648ee810e263591ecca7d"
cas_config_opid_index: -1
I20260925 10:08:28.815917 20609 consensus_queue.cc:237] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: NON_VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: true } }
I20260925 10:08:28.816483 21169 raft_consensus.cc:2932] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 86de7b8cfd2f42429708bdc3fa39e704 (127.14.41.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: NON_VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: true } } }
I20260925 10:08:28.816926 21115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:28.817698 21169 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.41.125:41713
I20260925 10:08:28.817972 21115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 1 FOLLOWER]: Refusing update from remote peer efc62ed2149648ee810e263591ecca7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 10:08:28.817997 21073 master.cc:537] Master@127.14.41.125:41713 shutting down...
I20260925 10:08:28.818118 21168 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "efc62ed2149648ee810e263591ecca7d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: NON_VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: true } } }
I20260925 10:08:28.818472 21169 consensus_queue.cc:1035] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: NON_VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:28.818626 21168 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: This master's current role is: LEADER
W20260925 10:08:28.819348 20593 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 10:08:28.819823 20593 consensus_peers.cc:487] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d -> Peer 86de7b8cfd2f42429708bdc3fa39e704 (127.14.41.125:41713): Couldn't send request to peer 86de7b8cfd2f42429708bdc3fa39e704. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 10:08:28.820986 21073 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:28.821089 21073 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:28.821142 21073 tablet_replica.cc:330] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: stopping tablet replica
I20260925 10:08:28.824007 21073 master.cc:552] Master@127.14.41.125:41713 shutdown complete.
I20260925 10:08:28.824082 21073 master_runner.cc:305] Clearing existing system tablet
I20260925 10:08:28.824222 21073 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:28.824286 21073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:28.824908 21073 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260925 10:08:28.825407 21173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:28.825534 21073 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:28.825591 21073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:28.825618 21073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/wal
uuid: "86de7b8cfd2f42429708bdc3fa39e704"
format_stamp: "Formatted at 2026-09-25 10:08:28 on dist-test-slave-s6tm"
server_key: "d8a6b70375099a6f959a1b8430916160"
server_key_iv: "0302daaeea471a10c29e9764eb46d0ab"
server_key_version: "encryptionkey@0"
I20260925 10:08:28.826284 21073 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:28.827896 21073 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 10:08:28.828051 21073 log.cc:1196] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 10:08:28.828251 21073 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: Deleting consensus metadata
I20260925 10:08:28.828382 21073 master_runner.cc:315] Copying system tablet from 127.14.41.126:43871
I20260925 10:08:28.829089 21073 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: tablet copy: Beginning tablet copy session from remote peer at address 127.14.41.126:43871
I20260925 10:08:28.836710 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.829521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38080 (local address 127.14.41.126:43871)
0925 10:08:28.829568 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:28.829571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:28.829619 (+ 48us) server_negotiation.cc:407] Connection header received
0925 10:08:28.829697 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:28.829698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:28.829708 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:28.829756 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:28.830130 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.830654 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.831853 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.832168 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.832243 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.833784 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:28.833786 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:28.833786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:28.833794 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:28.835435 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.835856 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.835857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.835857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.835915 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.836220 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.836221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.836221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.836341 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:28.836396 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:28.836572 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 10:08:28.836640 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:08:28.836781 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.829408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:38080)
0925 10:08:28.829543 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:28.829555 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:28.829630 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:28.829765 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:28.829765 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:28.829971 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:28.830038 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:28.830040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.830751 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.830752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:28.831611 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:28.831614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.832253 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.832255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:28.832337 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.833244 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:28.833250 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:28.833699 (+ 449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:28.835528 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:28.835529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:28.835530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:28.835775 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:28.836083 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:28.836084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:28.836084 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:28.836152 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:28.836409 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:28.836410 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:28.836469 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 10:08:28.836543 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 10:08:28.836662 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":88,"threads_started":1}
I20260925 10:08:28.837096 20629 tablet_copy_service.cc:140] P efc62ed2149648ee810e263591ecca7d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 86de7b8cfd2f42429708bdc3fa39e704 ({username='oryx', principal='oryx/127.14.41.125@KRBTEST.COM'} at 127.0.0.1:38080)
I20260925 10:08:28.837194 20629 tablet_copy_service.cc:161] P efc62ed2149648ee810e263591ecca7d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 86de7b8cfd2f42429708bdc3fa39e704 at {username='oryx', principal='oryx/127.14.41.125@KRBTEST.COM'} at 127.0.0.1:38080: session id = 86de7b8cfd2f42429708bdc3fa39e704-00000000000000000000000000000000
I20260925 10:08:28.837855 20629 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 10:08:28.838503 21073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 10:08:28.839735 21073 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: tablet copy: Starting download of 0 data blocks...
I20260925 10:08:28.839984 21073 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: tablet copy: Starting download of 1 WAL segments...
I20260925 10:08:28.841146 21073 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 10:08:28.842235 20629 tablet_copy_service.cc:342] P efc62ed2149648ee810e263591ecca7d: Request end of tablet copy session 86de7b8cfd2f42429708bdc3fa39e704-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.41.125@KRBTEST.COM'} at 127.0.0.1:38080
I20260925 10:08:28.842308 20629 tablet_copy_service.cc:433] P efc62ed2149648ee810e263591ecca7d: ending tablet copy session 86de7b8cfd2f42429708bdc3fa39e704-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 86de7b8cfd2f42429708bdc3fa39e704
I20260925 10:08:28.844992 21073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:28.846573 21180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:28.846611 21181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:28.846593 21183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:28.847095 21073 server_base.cc:1034] running on GCE node
I20260925 10:08:28.847199 21073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:28.847442 21073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:28.848601 21073 hybrid_clock.cc:648] HybridClock initialized: now 1790330908848591 us; error 26 us; skew 500 ppm
Sep 25 10:08:28 dist-test-slave-s6tm krb5kdc[20564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330908, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:28.850229 21073 init.cc:377] Logged in from keytab as oryx/127.14.41.125@KRBTEST.COM (short username oryx)
I20260925 10:08:28.850538 21073 webserver.cc:460] Webserver started at http://127.14.41.125:33297/ using document root <none> and password file <none>
I20260925 10:08:28.850739 21073 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:28.850783 21073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:28.851511 21073 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:08:28.852239 21190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:28.852442 21073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:08:28.852500 21073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/wal
uuid: "86de7b8cfd2f42429708bdc3fa39e704"
format_stamp: "Formatted at 2026-09-25 10:08:28 on dist-test-slave-s6tm"
server_key: "d8a6b70375099a6f959a1b8430916160"
server_key_iv: "0302daaeea471a10c29e9764eb46d0ab"
server_key_version: "encryptionkey@0"
I20260925 10:08:28.852564 21073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:28.873482 21073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:28.875344 21073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:28.878599 21073 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.125:41713
I20260925 10:08:28.879035 21073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-1/data/info.pb
I20260925 10:08:28.879154 21242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.125:41713 every 8 connection(s)
I20260925 10:08:28.879866 21243 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:08:28.880366 21243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: Bootstrap starting.
I20260925 10:08:28.883103 21243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:08:28.883378 21243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: Bootstrap complete.
I20260925 10:08:28.883599 21243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: NON_VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: true } }
I20260925 10:08:28.883728 21243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 86de7b8cfd2f42429708bdc3fa39e704, State: Initialized, Role: LEARNER
I20260925 10:08:28.883777 21243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: NON_VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: true } }
I20260925 10:08:28.884044 21243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:28.883997 21246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: NON_VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: true } } }
I20260925 10:08:28.884215 21246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:08:28.886003 21258 catalog_manager.cc:1259] Loaded cluster ID: 5b434835f51047d689e6ab46671765f6
I20260925 10:08:28.886090 21258 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: loading cluster ID for follower catalog manager: success
I20260925 10:08:28.887346 21258 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: acquiring CA information for follower catalog manager: success
I20260925 10:08:28.887708 21258 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:08:28.891878 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:28.884231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59898 (local address 127.14.41.125:41713)
0925 10:08:28.884391 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:28.884394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:28.884405 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:08:28.884436 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:28.884437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:28.884449 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:28.884545 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:28.885400 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.886453 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.887392 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:28.887673 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:28.887763 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:28.889102 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:28.889103 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:28.889104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:28.889110 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:28.890626 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.891031 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.891033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.891034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.891097 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:28.891302 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:28.891302 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:28.891303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:28.891418 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:28.891523 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:28.891654 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 10:08:28.891752 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for oryx/127.14.41.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.41.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.41.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.124.keytab.
Entry for principal oryx/127.14.41.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.124.keytab.
I20260925 10:08:28.924289 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.124:32911
--webserver_interface=127.14.41.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.124.keytab
--principal=oryx/127.14.41.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:43871,127.14.41.125:41713,127.14.41.124:32911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:29.016906 21261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:29.017131 21261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:29.017154 21261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:29.018298 21261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:29.018520 21261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:29.018566 21261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:29.018581 21261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:29.018594 21261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:08:29.020697 21261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:43871,127.14.41.125:41713,127.14.41.124:32911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.124.keytab
--principal=oryx/127.14.41.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.124:32911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.41.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:29.020932 21261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:29.021122 21261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:29.026222 21267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:29.026278 21266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:29.026510 21261 server_base.cc:1034] running on GCE node
W20260925 10:08:29.026521 21269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:29.026876 21261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:29.027210 21261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:29.028399 21261 hybrid_clock.cc:648] HybridClock initialized: now 1790330909028395 us; error 36 us; skew 500 ppm
Sep 25 10:08:29 dist-test-slave-s6tm krb5kdc[20564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330909, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:29.031031 21261 init.cc:377] Logged in from keytab as oryx/127.14.41.124@KRBTEST.COM (short username oryx)
I20260925 10:08:29.031529 21261 webserver.cc:460] Webserver started at http://127.14.41.124:40251/ using document root <none> and password file <none>
I20260925 10:08:29.031723 21261 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:29.031764 21261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:29.031862 21261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:29.033074 21261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/data/instance:
uuid: "7bd20a07dd664c12a0dcc7ddbab4e072"
format_stamp: "Formatted at 2026-09-25 10:08:29 on dist-test-slave-s6tm"
server_key: "c6f03d3ca0a1a79dfd979553a1482711"
server_key_iv: "aed224da4587d717ca882884491afee3"
server_key_version: "encryptionkey@0"
I20260925 10:08:29.033542 21261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/wal/instance:
uuid: "7bd20a07dd664c12a0dcc7ddbab4e072"
format_stamp: "Formatted at 2026-09-25 10:08:29 on dist-test-slave-s6tm"
server_key: "c6f03d3ca0a1a79dfd979553a1482711"
server_key_iv: "aed224da4587d717ca882884491afee3"
server_key_version: "encryptionkey@0"
I20260925 10:08:29.035074 21261 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:08:29.036000 21276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:29.036191 21261 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:29.036291 21261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/wal
uuid: "7bd20a07dd664c12a0dcc7ddbab4e072"
format_stamp: "Formatted at 2026-09-25 10:08:29 on dist-test-slave-s6tm"
server_key: "c6f03d3ca0a1a79dfd979553a1482711"
server_key_iv: "aed224da4587d717ca882884491afee3"
server_key_version: "encryptionkey@0"
I20260925 10:08:29.036412 21261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:29.046288 21261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:29.049242 21261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:29.049422 21261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:29.053635 21261 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.124:32911
I20260925 10:08:29.053834 21328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.124:32911 every 8 connection(s)
I20260925 10:08:29.054106 21261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/data/info.pb
I20260925 10:08:29.055109 21329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:29.056788 21329 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:08:29.060372 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 21261
Sep 25 10:08:29 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330909, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.124@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM
Sep 25 10:08:29 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.41.124@KRBTEST.COM
I20260925 10:08:29.067206 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.057609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38088 (local address 127.14.41.126:43871)
0925 10:08:29.057659 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.057663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.057774 (+ 111us) server_negotiation.cc:407] Connection header received
0925 10:08:29.058355 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.058356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.058366 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.058430 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.059015 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.059593 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.060748 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.061061 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.061143 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.063766 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.063768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.063769 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.063797 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.065613 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.066118 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.066119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.066120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.066190 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.066474 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.066475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.066476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.066618 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.066674 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.067056 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.067138 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:08:29.067507 21331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.057476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:38088)
0925 10:08:29.057643 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.057685 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.058270 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.058497 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.058500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.058749 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:29.058941 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.058945 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.059670 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.059672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.060654 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.060657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.061151 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.061152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.061274 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.061838 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:29.061849 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:29.063673 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:29.065704 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.065713 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.065716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.066024 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.066266 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.066267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.066267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.066379 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.066691 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:29.066697 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:29.066880 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.067016 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.067231 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260925 10:08:29.068135 21329 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:08:29.070436 21332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.060933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35018 (local address 127.14.41.124:32911)
0925 10:08:29.061129 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.061134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.061157 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:08:29.061180 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.061182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.061204 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.061278 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.062034 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.062900 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.063828 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.064287 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.064383 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.067084 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.067090 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.067090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.067096 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.069202 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.069563 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.069566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.069566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.069629 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.069835 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.069836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.069837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.069991 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.070141 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.070228 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.070308 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":106,"threads_started":1}
Sep 25 10:08:29 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330909, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.124@KRBTEST.COM for oryx/127.14.41.125@KRBTEST.COM
I20260925 10:08:29.075065 21331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.068453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:41713 (local address 127.0.0.1:59910)
0925 10:08:29.068585 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.068596 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.068648 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.068769 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.068770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.068975 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:29.069048 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.069051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.069853 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.069855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.070416 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.070418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.070895 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.070896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.070980 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.071407 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:29.071412 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:29.072617 (+ 1205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:29.074218 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.074219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.074220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.074380 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.074566 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.074567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.074567 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.074616 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.074823 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:29.074824 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:29.074861 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.074911 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.074998 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260925 10:08:29.075065 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.068533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59910 (local address 127.14.41.125:41713)
0925 10:08:29.068585 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.068587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.068634 (+ 47us) server_negotiation.cc:407] Connection header received
0925 10:08:29.068707 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.068708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.068717 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.068757 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.069130 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.069746 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.070512 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.070809 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.070895 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.072736 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.072737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.072738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.072745 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.074143 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.074452 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.074453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.074453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.074506 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.074669 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.074669 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.074670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.074772 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.074827 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.074925 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.074999 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 10:08:29.075657 21329 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:08:29 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330909, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.124@KRBTEST.COM for oryx/127.14.41.124@KRBTEST.COM
I20260925 10:08:29.081986 21331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.075921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:32911 (local address 127.0.0.1:35030)
0925 10:08:29.076025 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.076035 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.076087 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.076176 (+ 89us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.076177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.076283 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:29.076316 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.076317 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.076965 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.076966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.077418 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.077420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.077786 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.077787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.077827 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.078252 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:29.078256 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:29.079525 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:29.081125 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.081126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.081126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.081346 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.081514 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.081514 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.081515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.081566 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.081752 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:29.081752 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:29.081787 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.081838 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.081914 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 10:08:29.082048 21332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.076005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.14.41.124:32911)
0925 10:08:29.076037 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.076039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.076075 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:08:29.076123 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.076124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.076133 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.076174 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.076411 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.076870 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.077514 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.077712 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.077780 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.079614 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.079615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.079616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.079622 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.081044 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.081399 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.081400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.081400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.081450 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.081605 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.081606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.081606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.081701 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.081745 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.081855 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.081958 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":16}
I20260925 10:08:29.083561 21329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072: Bootstrap starting.
I20260925 10:08:29.084556 21329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:29.084888 21329 log.cc:824] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:29.085760 21329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072: No bootstrap required, opened a new log
I20260925 10:08:29.087247 21329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } }
I20260925 10:08:29.087402 21329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:29.087483 21329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd20a07dd664c12a0dcc7ddbab4e072, State: Initialized, Role: FOLLOWER
I20260925 10:08:29.087663 21329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } }
I20260925 10:08:29.088035 21329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:29.088008 21334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } } }
I20260925 10:08:29.088125 21334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:08:29.089848 21345 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:08:29.089913 21345 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:08:29.093117 21261 master_runner.cc:428] Detected that this master 7bd20a07dd664c12a0dcc7ddbab4e072 is joining an existing cluster
I20260925 10:08:29.093176 21261 master_runner.cc:432] Initiating AddMaster RPC to add 127.14.41.124:32911
I20260925 10:08:29.102272 21332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.094741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35034 (local address 127.14.41.124:32911)
0925 10:08:29.094779 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.094781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.095255 (+ 474us) server_negotiation.cc:407] Connection header received
0925 10:08:29.095338 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.095338 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.095346 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.095373 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.095770 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.096517 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.097539 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.097810 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.097849 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.099421 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.099423 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.099423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.099430 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.101009 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.101419 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.101420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.101421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.101477 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.101734 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.101735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.101736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.101904 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.101966 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.102126 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.102204 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":18}
I20260925 10:08:29.102406 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.094707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:32911 (local address 127.0.0.1:35034)
0925 10:08:29.095197 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.095206 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.095277 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.095425 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.095425 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.095631 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:29.095689 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.095691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.096609 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.096610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.097445 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.097447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.097956 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.097957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.098062 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.098599 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:29.098604 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:29.099106 (+ 502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:29.101097 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.101098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.101099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.101324 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.101563 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.101564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.101564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.101645 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.101982 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:29.101983 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:29.102029 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.102109 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.102293 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":75,"thread_start_us":79,"threads_started":1}
I20260925 10:08:29.122280 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 20671
I20260925 10:08:29.125552 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.094383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38092 (local address 127.14.41.126:43871)
0925 10:08:29.094455 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.094458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.094762 (+ 304us) server_negotiation.cc:407] Connection header received
0925 10:08:29.094864 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.094865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.094879 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.094941 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.095460 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.096283 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.097502 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.097813 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.097855 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.121476 (+ 23621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.121478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.121478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.121489 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.123971 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.124620 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.124621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.124622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.124701 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.124921 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.124923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.124923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.125077 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.125123 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.125357 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.125453 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 10:08:29.125597 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.094632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59920 (local address 127.14.41.125:41713)
0925 10:08:29.094828 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.094831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.094947 (+ 116us) server_negotiation.cc:407] Connection header received
0925 10:08:29.095047 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.095048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.095058 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.095114 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.095489 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.096292 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.097503 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.097813 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.097855 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.121405 (+ 23550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.121407 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.121408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.121418 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.123978 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.124620 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.124621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.124622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.124700 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.125039 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.125040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.125040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.125183 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.125225 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.125426 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.125507 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260925 10:08:29.125968 21352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.094299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:38092)
0925 10:08:29.094689 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.094701 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.094791 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.094965 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.094966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.095255 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:29.095358 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.095360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.096387 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.096388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.097396 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.097399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.097953 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.097954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.098047 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.120701 (+ 22654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:29.120707 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:29.121242 (+ 535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:29.124098 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.124099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.124100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.124524 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.124779 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.124780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.124780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.124842 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.125169 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:29.125170 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:29.125218 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.125700 (+ 482us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.125855 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":78,"thread_start_us":117,"threads_started":1}
I20260925 10:08:29.126964 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.094539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:41713 (local address 127.0.0.1:59920)
0925 10:08:29.094894 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.094904 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.094971 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.095120 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.095120 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.095351 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:29.095400 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.095402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.096387 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.096388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.097396 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.097399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.097954 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.097955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.098026 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.120539 (+ 22513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:29.120546 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:29.121242 (+ 696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:29.124098 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.124099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.124100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.124519 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.124909 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.124909 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.124910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.124966 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.125271 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:29.125272 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:29.125308 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.126716 (+ 1408us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.126845 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":41,"thread_start_us":75,"threads_started":1}
I20260925 10:08:29.131120 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:40287
--local_ip_for_outbound_sockets=127.14.41.65
--tserver_master_addrs=127.14.41.126:43871
--webserver_port=35207
--webserver_interface=127.14.41.65
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.65.keytab
--principal=oryx/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.41.126:43871,127.14.41.125:41713
--tserver_master_addrs=127.14.41.126:43871,127.14.41.125:41713,127.14.41.124:32911 with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:08:29 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330908, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.126@KRBTEST.COM for oryx/127.14.41.124@KRBTEST.COM
I20260925 10:08:29.138055 21332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.128762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35050 (local address 127.14.41.124:32911)
0925 10:08:29.128802 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.128804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.128847 (+ 43us) server_negotiation.cc:407] Connection header received
0925 10:08:29.128935 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.128936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.128946 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.129001 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.129300 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.129783 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.130511 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.130792 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.130868 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.134826 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.134828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.134829 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.134837 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.136678 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.137155 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.137157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.137158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.137220 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.137456 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.137456 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.137457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.137596 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.137665 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.137864 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.137954 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 10:08:29.138113 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.128720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:32911 (local address 127.0.0.1:35050)
0925 10:08:29.128804 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.128815 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.128864 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.129006 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.129006 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.129168 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:29.129212 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.129214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.129868 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.129869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.130414 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.130416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.132438 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.132440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.132527 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.133019 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:29.133026 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:29.134714 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:29.136792 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.136793 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.136794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.137066 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.137301 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.137302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.137303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.137379 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.137673 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:29.137674 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:29.137733 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.137824 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.137961 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 10:08:29.138685 20609 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.14.41.124:32911: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072"
member_type: NON_VOTER
last_known_addr {
host: "127.14.41.124"
port: 32911
}
attrs {
promote: true
}
}
dest_uuid: "efc62ed2149648ee810e263591ecca7d"
cas_config_opid_index: 5
I20260925 10:08:29.138988 20609 consensus_queue.cc:237] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: NON_VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: true } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: NON_VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: true } }
I20260925 10:08:29.139384 21168 raft_consensus.cc:2932] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7bd20a07dd664c12a0dcc7ddbab4e072 (127.14.41.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: NON_VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: true } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: NON_VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: true } } }
I20260925 10:08:29.140735 21168 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.14.41.124:32911
I20260925 10:08:29.140836 21169 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "efc62ed2149648ee810e263591ecca7d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: NON_VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: true } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: NON_VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: true } } }
I20260925 10:08:29.140946 21169 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:29.141111 21303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:29.142297 21303 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 1 FOLLOWER]: Refusing update from remote peer efc62ed2149648ee810e263591ecca7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 10:08:29.142616 21169 consensus_queue.cc:1035] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: NON_VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:29.142887 21261 master.cc:537] Master@127.14.41.124:32911 shutting down...
W20260925 10:08:29.143373 20593 consensus_peers.cc:487] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d -> Peer 7bd20a07dd664c12a0dcc7ddbab4e072 (127.14.41.124:32911): Couldn't send request to peer 7bd20a07dd664c12a0dcc7ddbab4e072. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 10:08:29.146489 21261 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:29.146593 21261 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:29.146636 21261 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072: stopping tablet replica
I20260925 10:08:29.147859 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.140874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59926 (local address 127.14.41.125:41713)
0925 10:08:29.142502 (+ 1628us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.142504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.142516 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:08:29.142544 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.142546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.142573 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.142653 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:29.143566 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.144596 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.146492 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.147123 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.147172 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.147254 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.147734 (+ 480us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.147785 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1596}
I20260925 10:08:29.148547 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.139477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:41713 (local address 127.0.0.1:59926)
0925 10:08:29.139686 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.139697 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.139732 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.143355 (+ 3623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.143356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.143359 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:29.143405 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.143406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.144706 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.144707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.146178 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.146181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.147542 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.147543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.147620 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.147653 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.148134 (+ 481us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.148399 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260925 10:08:29.148960 21217 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 1 LEARNER]: Refusing update from remote peer efc62ed2149648ee810e263591ecca7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 10:08:29.149210 21169 consensus_queue.cc:1035] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: NON_VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 10:08:29.149780 21217 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7bd20a07dd664c12a0dcc7ddbab4e072 (127.14.41.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: NON_VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: true } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: NON_VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: true } } }
I20260925 10:08:29.151196 21246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efc62ed2149648ee810e263591ecca7d. Latest consensus state: current_term: 1 leader_uuid: "efc62ed2149648ee810e263591ecca7d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: NON_VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: true } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: NON_VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: true } } }
I20260925 10:08:29.151304 21246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:08:29.151350 21246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efc62ed2149648ee810e263591ecca7d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: NON_VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: true } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: NON_VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: true } } }
I20260925 10:08:29.151392 21246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:08:29.151628 21169 raft_consensus.cc:1050] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d: attempting to promote NON_VOTER 86de7b8cfd2f42429708bdc3fa39e704 to VOTER
I20260925 10:08:29.151851 21169 consensus_queue.cc:237] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: NON_VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: true } }
I20260925 10:08:29.152468 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.152333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:32911 (local address 127.0.0.1:35066)
0925 10:08:29.152387 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.152438 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 10:08:29.152503 21217 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 1 LEARNER]: Refusing update from remote peer efc62ed2149648ee810e263591ecca7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 10:08:29.152827 21169 consensus_queue.cc:1035] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 10:08:29.153110 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.152935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:32911 (local address 127.0.0.1:35068)
0925 10:08:29.153049 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.153074 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260925 10:08:29.153221 20593 consensus_peers.cc:487] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d -> Peer 7bd20a07dd664c12a0dcc7ddbab4e072 (127.14.41.124:32911): Couldn't send request to peer 7bd20a07dd664c12a0dcc7ddbab4e072. Status: Network error: Client connection negotiation failed: client connection to 127.14.41.124:32911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 10:08:29.153430 21169 raft_consensus.cc:2932] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 86de7b8cfd2f42429708bdc3fa39e704 (127.14.41.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: NON_VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: true } } }
I20260925 10:08:29.153584 21217 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 86de7b8cfd2f42429708bdc3fa39e704 (127.14.41.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: NON_VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: true } } }
I20260925 10:08:29.154357 21169 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "efc62ed2149648ee810e263591ecca7d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: NON_VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: true } } }
I20260925 10:08:29.154430 21169 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:29.154477 21169 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efc62ed2149648ee810e263591ecca7d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: NON_VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: true } } }
I20260925 10:08:29.154516 21169 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:29.154469 21246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efc62ed2149648ee810e263591ecca7d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: NON_VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: true } } }
I20260925 10:08:29.154543 21246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:08:29.160707 21261 master.cc:552] Master@127.14.41.124:32911 shutdown complete.
I20260925 10:08:29.160770 21261 master_runner.cc:305] Clearing existing system tablet
I20260925 10:08:29.160920 21261 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:29.160972 21261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:29.161768 21261 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:08:29.162387 21359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:29.162611 21261 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:29.162675 21261 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:29.162699 21261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/wal
uuid: "7bd20a07dd664c12a0dcc7ddbab4e072"
format_stamp: "Formatted at 2026-09-25 10:08:29 on dist-test-slave-s6tm"
server_key: "c6f03d3ca0a1a79dfd979553a1482711"
server_key_iv: "aed224da4587d717ca882884491afee3"
server_key_version: "encryptionkey@0"
I20260925 10:08:29.163472 21261 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:29.165345 21261 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 10:08:29.165421 21261 log.cc:1196] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 10:08:29.165632 21261 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072: Deleting consensus metadata
I20260925 10:08:29.165954 21261 master_runner.cc:315] Copying system tablet from 127.14.41.126:43871
I20260925 10:08:29.166590 21261 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072: tablet copy: Beginning tablet copy session from remote peer at address 127.14.41.126:43871
I20260925 10:08:29.174364 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.166992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38096 (local address 127.14.41.126:43871)
0925 10:08:29.167048 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.167050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.167160 (+ 110us) server_negotiation.cc:407] Connection header received
0925 10:08:29.167222 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.167223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.167241 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.167303 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.167691 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.168520 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.169583 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.169884 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.169960 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.171198 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.171200 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.171200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.171208 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.172999 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.173420 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.173421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.173422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.173492 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.173713 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.173714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.173715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.173860 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.173928 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.174181 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.174273 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:08:29.174373 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.166901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:38096)
0925 10:08:29.167066 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.167078 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.167161 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.167327 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.167328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.167529 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:29.167600 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.167603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.168625 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.168626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.169474 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.169478 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.169979 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.169980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.170052 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.170626 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:29.170632 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:29.171092 (+ 460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:29.173095 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.173096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.173097 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.173337 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.173567 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.173568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.173569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.173640 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.173990 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:29.173991 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:29.174055 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.174139 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.174261 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":72,"threads_started":1}
I20260925 10:08:29.175027 20629 tablet_copy_service.cc:140] P efc62ed2149648ee810e263591ecca7d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7bd20a07dd664c12a0dcc7ddbab4e072 ({username='oryx', principal='oryx/127.14.41.124@KRBTEST.COM'} at 127.0.0.1:38096)
I20260925 10:08:29.175091 20629 tablet_copy_service.cc:161] P efc62ed2149648ee810e263591ecca7d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7bd20a07dd664c12a0dcc7ddbab4e072 at {username='oryx', principal='oryx/127.14.41.124@KRBTEST.COM'} at 127.0.0.1:38096: session id = 7bd20a07dd664c12a0dcc7ddbab4e072-00000000000000000000000000000000
I20260925 10:08:29.175766 20629 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 10:08:29.176395 21261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 10:08:29.177744 21261 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072: tablet copy: Starting download of 0 data blocks...
I20260925 10:08:29.177896 21261 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072: tablet copy: Starting download of 1 WAL segments...
I20260925 10:08:29.178645 21261 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 10:08:29.179601 20629 tablet_copy_service.cc:342] P efc62ed2149648ee810e263591ecca7d: Request end of tablet copy session 7bd20a07dd664c12a0dcc7ddbab4e072-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.14.41.124@KRBTEST.COM'} at 127.0.0.1:38096
I20260925 10:08:29.179656 20629 tablet_copy_service.cc:433] P efc62ed2149648ee810e263591ecca7d: ending tablet copy session 7bd20a07dd664c12a0dcc7ddbab4e072-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7bd20a07dd664c12a0dcc7ddbab4e072
I20260925 10:08:29.182541 21261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:29.184171 21366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:29.184922 21367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:29.185222 21369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:29.185389 21261 server_base.cc:1034] running on GCE node
I20260925 10:08:29.185490 21261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:29.185639 21261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:29.186789 21261 hybrid_clock.cc:648] HybridClock initialized: now 1790330909186782 us; error 25 us; skew 500 ppm
Sep 25 10:08:29 dist-test-slave-s6tm krb5kdc[20564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330909, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:29.188474 21261 init.cc:377] Logged in from keytab as oryx/127.14.41.124@KRBTEST.COM (short username oryx)
I20260925 10:08:29.188776 21261 webserver.cc:460] Webserver started at http://127.14.41.124:40881/ using document root <none> and password file <none>
I20260925 10:08:29.188905 21261 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:29.188954 21261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:29.189743 21261 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:29.190377 21376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:29.190553 21261 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:08:29.190614 21261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/wal
uuid: "7bd20a07dd664c12a0dcc7ddbab4e072"
format_stamp: "Formatted at 2026-09-25 10:08:29 on dist-test-slave-s6tm"
server_key: "c6f03d3ca0a1a79dfd979553a1482711"
server_key_iv: "aed224da4587d717ca882884491afee3"
server_key_version: "encryptionkey@0"
I20260925 10:08:29.190683 21261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:29.206164 21261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:29.208318 21261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:29.211587 21261 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.124:32911
I20260925 10:08:29.212013 21261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/master-2/data/info.pb
I20260925 10:08:29.213083 21429 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:08:29.213354 21428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.124:32911 every 8 connection(s)
I20260925 10:08:29.213610 21429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072: Bootstrap starting.
I20260925 10:08:29.216761 21429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:08:29.217031 21429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072: Bootstrap complete.
I20260925 10:08:29.217260 21429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: NON_VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: true } }
I20260925 10:08:29.217360 21429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7bd20a07dd664c12a0dcc7ddbab4e072, State: Initialized, Role: LEARNER
I20260925 10:08:29.217401 21429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: NON_VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: true } }
I20260925 10:08:29.217644 21429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:29.217636 21432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: NON_VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: true } } }
I20260925 10:08:29.217921 21432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:08:29.219190 21443 catalog_manager.cc:1259] Loaded cluster ID: 5b434835f51047d689e6ab46671765f6
I20260925 10:08:29.219308 21443 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072: loading cluster ID for follower catalog manager: success
I20260925 10:08:29.220543 21443 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072: acquiring CA information for follower catalog manager: success
I20260925 10:08:29.220863 21443 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 10:08:29.236929 21354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:29.237156 21354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:29.237221 21354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:29.238487 21354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:29.238535 21354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:29.238564 21354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:08:29.240342 21354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.65.keytab
--principal=oryx/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:40287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=35207
--tserver_master_addrs=127.14.41.126:43871,127.14.41.125:41713,127.14.41.124:32911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:29.240612 21354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:29.240864 21354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:29.247021 21449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:29.247022 21448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:29.247033 21451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:29.247335 21354 server_base.cc:1034] running on GCE node
I20260925 10:08:29.247598 21354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:29.247846 21354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:29.249030 21354 hybrid_clock.cc:648] HybridClock initialized: now 1790330909249007 us; error 36 us; skew 500 ppm
Sep 25 10:08:29 dist-test-slave-s6tm krb5kdc[20564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330909, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:29.251716 21354 init.cc:377] Logged in from keytab as oryx/127.14.41.65@KRBTEST.COM (short username oryx)
I20260925 10:08:29.252234 21354 webserver.cc:460] Webserver started at http://127.14.41.65:35207/ using document root <none> and password file <none>
I20260925 10:08:29.252487 21354 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:29.252574 21354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:29.254109 21354 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:08:29.254801 21458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:29.254992 21354 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:29.255111 21354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "e02b6aac6c8f403396d2a6cfe768d541"
format_stamp: "Formatted at 2026-09-25 10:08:28 on dist-test-slave-s6tm"
server_key: "9bd8d8ec78d0bc120419b51d2fc224f1"
server_key_iv: "3cbef466bfc27301059d1e4ebb402c93"
server_key_version: "encryptionkey@0"
I20260925 10:08:29.255527 21354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:29.273746 21354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:29.276443 21354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:29.276625 21354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:29.276877 21354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:29.277242 21354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:29.277298 21354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:29.277339 21354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:29.277381 21354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:29.283589 21354 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:40287
I20260925 10:08:29.283632 21571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:40287 every 8 connection(s)
I20260925 10:08:29.284063 21354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:08:29.288134 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 21354
I20260925 10:08:29.288298 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 20805
Sep 25 10:08:29 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330909, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.124@KRBTEST.COM
Sep 25 10:08:29 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330909, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM
I20260925 10:08:29.295135 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:44663
--local_ip_for_outbound_sockets=127.14.41.66
--tserver_master_addrs=127.14.41.126:43871
--webserver_port=46717
--webserver_interface=127.14.41.66
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.66.keytab
--principal=oryx/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.41.126:43871,127.14.41.125:41713
--tserver_master_addrs=127.14.41.126:43871,127.14.41.125:41713,127.14.41.124:32911 with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:08:29 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330909, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.125@KRBTEST.COM
I20260925 10:08:29.296561 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.285171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:32911 (local address 127.14.41.65:43129)
0925 10:08:29.285312 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.285323 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.285769 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.286052 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.286053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.286414 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:29.286610 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.286616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.287826 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.287827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.288548 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.288550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.289023 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.289024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.289167 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.289747 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:29.289757 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:29.292267 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:29.294665 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.294674 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.294676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.295042 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.295277 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.295278 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.295278 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.295428 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.295733 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:29.295739 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:29.295968 (+ 229us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.296065 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.296261 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":57,"threads_started":1}
I20260925 10:08:29.296984 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.285579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:43129 (local address 127.14.41.124:32911)
0925 10:08:29.285688 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.285690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.285699 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:08:29.285918 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.285919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.285933 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.286028 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.286731 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.287740 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.288674 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.288933 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.289411 (+ 478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.292387 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.292388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.292388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.292397 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.294569 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.295136 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.295137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.295137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.295197 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.295508 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.295508 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.295509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.295660 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.296724 (+ 1064us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.296770 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.296873 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":60,"threads_started":1}
I20260925 10:08:29.297454 21572 heartbeater.cc:347] Connected to a master server at 127.14.41.124:32911
I20260925 10:08:29.297509 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.285571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:59683 (local address 127.14.41.126:43871)
0925 10:08:29.285643 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.285645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.285656 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:08:29.285926 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.285928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.285939 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.285992 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.286714 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.287229 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.288826 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.289154 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.289216 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.292442 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.292444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.292445 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.292454 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.294795 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.295525 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.295527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.295527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.295591 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.295808 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.295809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.295810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.295976 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.296357 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.297202 (+ 845us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.297390 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 10:08:29.297565 21572 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:29.297775 21572 heartbeater.cc:511] Master 127.14.41.124:32911 requested a full tablet report, sending...
I20260925 10:08:29.297782 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.285334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.65:59683)
0925 10:08:29.285446 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.285456 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.285773 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.286044 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.286045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.286402 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:29.286610 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.286615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.287343 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.287345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.288493 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.288497 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.289275 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.289276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.289345 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.290103 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:29.290108 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:29.292286 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:29.294890 (+ 2604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.294891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.294891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.295152 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.295678 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.295679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.295679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.295741 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.296398 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:29.296398 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:29.296452 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.296524 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.297657 (+ 1133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":65,"threads_started":1}
I20260925 10:08:29.298418 21573 heartbeater.cc:347] Connected to a master server at 127.14.41.126:43871
I20260925 10:08:29.298467 21573 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:29.298471 21393 ts_manager.cc:194] Registered new tserver with Master: e02b6aac6c8f403396d2a6cfe768d541 (127.14.41.65:40287)
I20260925 10:08:29.298532 21573 heartbeater.cc:511] Master 127.14.41.126:43871 requested a full tablet report, sending...
I20260925 10:08:29.298877 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.284869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:38479 (local address 127.14.41.125:41713)
0925 10:08:29.284914 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.284917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.285158 (+ 241us) server_negotiation.cc:407] Connection header received
0925 10:08:29.285863 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.285865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.285877 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.285936 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.286751 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.287523 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.292772 (+ 5249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.293095 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.293164 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.295782 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.295784 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.295784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.295792 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.297551 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.297966 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.297968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.297968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.298028 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.298239 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.298240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.298240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.298386 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.298431 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.298695 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.298791 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 10:08:29.298923 20609 ts_manager.cc:194] Re-registered known tserver with Master: e02b6aac6c8f403396d2a6cfe768d541 (127.14.41.65:40287)
I20260925 10:08:29.299111 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.284856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:41713 (local address 127.14.41.65:38479)
0925 10:08:29.285037 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.285083 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.285751 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.286021 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.286027 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.286606 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:29.286659 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.286661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.287617 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.287618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.292659 (+ 5041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.292662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.293538 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.293540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.293622 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.294120 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:29.294125 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:29.295665 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:29.297641 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.297642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.297643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.297866 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.298101 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.298102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.298102 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.298158 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.298479 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:29.298480 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:29.298555 (+ 75us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.298651 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.298758 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":108,"threads_started":1}
I20260925 10:08:29.299430 20609 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:59683
I20260925 10:08:29.299564 21574 heartbeater.cc:347] Connected to a master server at 127.14.41.125:41713
I20260925 10:08:29.299603 21574 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:29.299652 21574 heartbeater.cc:511] Master 127.14.41.125:41713 requested a full tablet report, sending...
I20260925 10:08:29.300069 21206 ts_manager.cc:194] Registered new tserver with Master: e02b6aac6c8f403396d2a6cfe768d541 (127.14.41.65:40287)
W20260925 10:08:29.383605 21580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:29.383848 21580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:29.383874 21580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:29.385277 21580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:29.385332 21580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:29.385363 21580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:08:29.386996 21580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.66.keytab
--principal=oryx/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:44663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=46717
--tserver_master_addrs=127.14.41.126:43871,127.14.41.125:41713,127.14.41.124:32911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:29.387223 21580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:29.387427 21580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:29.393169 21586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:29.393261 21588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:29.393178 21585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:29.393651 21580 server_base.cc:1034] running on GCE node
I20260925 10:08:29.393991 21580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:29.394294 21580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:29.395511 21580 hybrid_clock.cc:648] HybridClock initialized: now 1790330909395499 us; error 41 us; skew 500 ppm
Sep 25 10:08:29 dist-test-slave-s6tm krb5kdc[20564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330909, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:29.398604 21580 init.cc:377] Logged in from keytab as oryx/127.14.41.66@KRBTEST.COM (short username oryx)
I20260925 10:08:29.399187 21580 webserver.cc:460] Webserver started at http://127.14.41.66:46717/ using document root <none> and password file <none>
I20260925 10:08:29.399464 21580 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:29.399576 21580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:29.401186 21580 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:29.402045 21595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:29.402278 21580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:29.402397 21580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "949ae6782e9d46b59871d4d1f6a05607"
format_stamp: "Formatted at 2026-09-25 10:08:28 on dist-test-slave-s6tm"
server_key: "03ff6fa824d81ffaf8ca3b4e3e722a1e"
server_key_iv: "e823724dc754bec211f2d528a2be206d"
server_key_version: "encryptionkey@0"
I20260925 10:08:29.402889 21580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:29.410655 21580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:29.413239 21580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:29.413429 21580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:29.413697 21580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:29.414032 21580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:29.414090 21580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:29.414134 21580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:29.414165 21580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:29.420722 21580 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:44663
I20260925 10:08:29.420751 21708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:44663 every 8 connection(s)
I20260925 10:08:29.421144 21580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:08:29.423672 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 21580
I20260925 10:08:29.423836 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 20939
Sep 25 10:08:29 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330909, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.124@KRBTEST.COM
I20260925 10:08:29.429781 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:44021
--local_ip_for_outbound_sockets=127.14.41.67
--tserver_master_addrs=127.14.41.126:43871
--webserver_port=46291
--webserver_interface=127.14.41.67
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.67.keytab
--principal=oryx/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.14.41.126:43871,127.14.41.125:41713
--tserver_master_addrs=127.14.41.126:43871,127.14.41.125:41713,127.14.41.124:32911 with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:08:29 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330909, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM
Sep 25 10:08:29 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330909, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.125@KRBTEST.COM
I20260925 10:08:29.435441 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.422228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:32911 (local address 127.14.41.66:49911)
0925 10:08:29.422539 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.422548 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.422920 (+ 372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.423288 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.423289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.423879 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:29.424195 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.424202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.425338 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.425339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.426566 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.426569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.426977 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.426978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.427117 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.427976 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:29.427989 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:29.430478 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:29.433088 (+ 2610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.433099 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.433102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.433522 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.433814 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.433815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.433815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.433981 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.434502 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:29.434510 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:29.434702 (+ 192us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.434941 (+ 239us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.435142 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":62,"threads_started":1}
I20260925 10:08:29.435840 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.421932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:51291 (local address 127.14.41.126:43871)
0925 10:08:29.422057 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.422059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.422240 (+ 181us) server_negotiation.cc:407] Connection header received
0925 10:08:29.423047 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.423049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.423065 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.423108 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.425471 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.426261 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.427498 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.427770 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.427826 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.431908 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.431910 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.431910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.431921 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.434439 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.434864 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.434865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.434865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.434927 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.435146 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.435146 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.435147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.435313 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.435362 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.435610 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.435703 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 10:08:29.436082 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.422330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:49911 (local address 127.14.41.124:32911)
0925 10:08:29.422392 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.422393 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.422615 (+ 222us) server_negotiation.cc:407] Connection header received
0925 10:08:29.423181 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.423182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.423195 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.423266 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.424679 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.425205 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.426660 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.426877 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.426963 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.430578 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.430580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.430580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.430589 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.433015 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.433606 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.433608 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.433609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.433675 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.434060 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.434062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.434062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.434389 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.434468 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.435848 (+ 1380us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.435988 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:08:29.436390 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.422462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:41289 (local address 127.14.41.125:41713)
0925 10:08:29.422504 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.422505 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.422756 (+ 251us) server_negotiation.cc:407] Connection header received
0925 10:08:29.423056 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.423057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.423072 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.423122 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.428453 (+ 5331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.429237 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.430290 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.430614 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.430691 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.433333 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.433335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.433335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.433344 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.435043 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.435452 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.435453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.435454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.435516 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.435716 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.435717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.435717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.435851 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.435894 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.436183 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.436295 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:08:29.436213 21713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.421921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.66:51291)
0925 10:08:29.422077 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.422121 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.422912 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.423219 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.423223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.423864 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:29.425372 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.425375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.426367 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.426368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.427406 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.427408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.427854 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.427856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.427975 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.428425 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:29.428430 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:29.431708 (+ 3278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:29.434530 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.434531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.434532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.434766 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.435008 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.435009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.435009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.435073 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.435420 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:29.435421 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:29.435470 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.435954 (+ 484us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.436078 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260925 10:08:29.436719 21715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.422393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:41713 (local address 127.14.41.66:41289)
0925 10:08:29.422673 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.422683 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.422920 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.423218 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.423223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.423864 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:29.428059 (+ 4195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.428062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.429341 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.429342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.430196 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.430199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.430699 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.430700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.430774 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.431453 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:29.431459 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:29.433219 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:29.435127 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.435128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.435128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.435357 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.435601 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.435602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.435602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.435654 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.436024 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:29.436025 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:29.436065 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.436493 (+ 428us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.436619 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":57,"threads_started":1}
I20260925 10:08:29.438005 21709 heartbeater.cc:347] Connected to a master server at 127.14.41.124:32911
I20260925 10:08:29.438115 21709 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:29.438292 21710 heartbeater.cc:347] Connected to a master server at 127.14.41.126:43871
I20260925 10:08:29.438335 21710 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:29.438340 21709 heartbeater.cc:511] Master 127.14.41.124:32911 requested a full tablet report, sending...
I20260925 10:08:29.438400 21710 heartbeater.cc:511] Master 127.14.41.126:43871 requested a full tablet report, sending...
I20260925 10:08:29.438731 21711 heartbeater.cc:347] Connected to a master server at 127.14.41.125:41713
I20260925 10:08:29.438778 21711 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:29.438834 21711 heartbeater.cc:511] Master 127.14.41.125:41713 requested a full tablet report, sending...
I20260925 10:08:29.439054 21393 ts_manager.cc:194] Registered new tserver with Master: 949ae6782e9d46b59871d4d1f6a05607 (127.14.41.66:44663)
I20260925 10:08:29.439172 21206 ts_manager.cc:194] Registered new tserver with Master: 949ae6782e9d46b59871d4d1f6a05607 (127.14.41.66:44663)
I20260925 10:08:29.439283 20609 ts_manager.cc:194] Re-registered known tserver with Master: 949ae6782e9d46b59871d4d1f6a05607 (127.14.41.66:44663)
I20260925 10:08:29.439836 20609 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:51291
W20260925 10:08:29.518711 21716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:29.518956 21716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:29.519030 21716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:29.520336 21716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:29.520411 21716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:29.520458 21716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:08:29.522006 21716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.67.keytab
--principal=oryx/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:44021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=46291
--tserver_master_addrs=127.14.41.126:43871,127.14.41.125:41713,127.14.41.124:32911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:29.522301 21716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:29.522555 21716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:29.527916 21721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:29.527915 21724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:29.528017 21722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:29.528144 21716 server_base.cc:1034] running on GCE node
I20260925 10:08:29.528380 21716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:29.528692 21716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:29.529940 21716 hybrid_clock.cc:648] HybridClock initialized: now 1790330909529920 us; error 32 us; skew 500 ppm
Sep 25 10:08:29 dist-test-slave-s6tm krb5kdc[20564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330909, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:29.532524 21716 init.cc:377] Logged in from keytab as oryx/127.14.41.67@KRBTEST.COM (short username oryx)
I20260925 10:08:29.533088 21716 webserver.cc:460] Webserver started at http://127.14.41.67:46291/ using document root <none> and password file <none>
I20260925 10:08:29.533306 21716 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:29.533365 21716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:29.535063 21716 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:08:29.535781 21731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:29.535971 21716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:29.536064 21716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "bb35bd8e8a9f42b1aa31926ca80f711b"
format_stamp: "Formatted at 2026-09-25 10:08:28 on dist-test-slave-s6tm"
server_key: "2e21832034ebc12b2825cecb739a8ffb"
server_key_iv: "bb4ed37b5f3e8aae2d042c342acccde2"
server_key_version: "encryptionkey@0"
I20260925 10:08:29.536533 21716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:29.537573 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.534168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:32911 (local address 127.0.0.1:35072)
0925 10:08:29.534273 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.534286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.534348 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.534559 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.534560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.534563 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:29.534619 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.534621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.535544 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.535545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.536632 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.536637 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.537283 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.537285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.537367 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.537371 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.537447 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.537501 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 10:08:29.537582 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.534336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35072 (local address 127.14.41.124:32911)
0925 10:08:29.534398 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.534400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.534408 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:08:29.534442 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.534443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.534465 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.534555 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:29.534675 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.535400 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.536764 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.537185 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.537247 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.537297 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.537493 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.537531 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 10:08:29.537979 21403 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 10:08:29.538152 21432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efc62ed2149648ee810e263591ecca7d. Latest consensus state: current_term: 1 leader_uuid: "efc62ed2149648ee810e263591ecca7d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: NON_VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: true } } }
I20260925 10:08:29.538275 21432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:08:29.555105 21716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:29.557616 21716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:29.557796 21716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:29.558072 21716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:29.558513 21716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:29.558552 21716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:29.558604 21716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:29.558627 21716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:29.564558 21716 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:44021
I20260925 10:08:29.564659 21844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:44021 every 8 connection(s)
I20260925 10:08:29.565023 21716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:08:29.566939 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 21716
Sep 25 10:08:29 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330909, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.124@KRBTEST.COM
Sep 25 10:08:29 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330909, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.125@KRBTEST.COM
Sep 25 10:08:29 dist-test-slave-s6tm krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330909, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM
I20260925 10:08:29.578830 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.566210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:33315 (local address 127.14.41.124:32911)
0925 10:08:29.566268 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.566270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.566285 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:08:29.566989 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.566991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.567010 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.567054 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.568091 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.568803 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.570383 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.570680 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.570724 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.575163 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.575164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.575164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.575171 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.577107 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.577649 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.577650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.577651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.577711 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.578038 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.578039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.578040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.578187 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.578260 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.578634 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.578720 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 10:08:29.579564 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.566135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:32911 (local address 127.14.41.67:33315)
0925 10:08:29.566239 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.566251 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.566758 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.567139 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.567139 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.567674 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:29.567973 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.567979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.568946 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.568947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.570277 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.570279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.570822 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.570822 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.570980 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.571662 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:29.571673 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:29.575049 (+ 3376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:29.577193 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.577204 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.577206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.577549 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.577791 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.577791 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.577792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.577968 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.578267 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:29.578274 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:29.578456 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.579081 (+ 625us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.579291 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":60,"thread_start_us":66,"threads_started":1}
I20260925 10:08:29.579872 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.565853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:45811 (local address 127.14.41.125:41713)
0925 10:08:29.565922 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.565924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.566151 (+ 227us) server_negotiation.cc:407] Connection header received
0925 10:08:29.566886 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.566887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.566900 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.566934 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.568100 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.568857 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.570384 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.570691 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.570728 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.576841 (+ 6113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.576843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.576843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.576851 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.578639 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.579014 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.579016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.579016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.579077 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.579279 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.579280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.579281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.579409 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.579470 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.579697 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.579783 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:08:29.580188 21845 heartbeater.cc:347] Connected to a master server at 127.14.41.124:32911
I20260925 10:08:29.580320 21845 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:29.580462 21849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.565873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:41713 (local address 127.14.41.67:45811)
0925 10:08:29.566021 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.566059 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.566756 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.567107 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.567112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.567644 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:29.567973 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.567979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.568948 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.568948 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.570277 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.570279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.570819 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.570820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.570980 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.571772 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:29.571777 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:29.576732 (+ 4955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:29.578718 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.578719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.578719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.578921 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.579152 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.579153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.579153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.579208 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.579523 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:29.579524 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:29.579565 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.580057 (+ 492us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.580164 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":56,"thread_start_us":67,"threads_started":1}
I20260925 10:08:29.580539 21845 heartbeater.cc:511] Master 127.14.41.124:32911 requested a full tablet report, sending...
I20260925 10:08:29.580965 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.566476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:40163 (local address 127.14.41.126:43871)
0925 10:08:29.566519 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.566521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.566530 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:08:29.566891 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.566893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.566907 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.566940 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.568967 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.569746 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.571298 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.571594 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.571653 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.576175 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.576177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.576177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.576186 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.579073 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.579828 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.579829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.579830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.579895 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.580151 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.580152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.580153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.580300 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.580368 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.580796 (+ 428us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.580873 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:08:29.581368 21393 ts_manager.cc:194] Registered new tserver with Master: bb35bd8e8a9f42b1aa31926ca80f711b (127.14.41.67:44021)
I20260925 10:08:29.581548 21847 heartbeater.cc:347] Connected to a master server at 127.14.41.125:41713
I20260925 10:08:29.580716 21851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.566310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.67:40163)
0925 10:08:29.566422 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.566431 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.566779 (+ 348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.567107 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.567111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.567856 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:29.567974 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.567981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.570129 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.570130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.571034 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.571037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.571721 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.571722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.571797 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.572197 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:29.572201 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:29.576078 (+ 3877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:29.579374 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.579376 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.579377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.579609 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.580018 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:29.580019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:29.580020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:29.580083 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:29.580395 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:29.580396 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:29.580437 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.580499 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.580596 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20260925 10:08:29.581595 21847 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:29.581663 21847 heartbeater.cc:511] Master 127.14.41.125:41713 requested a full tablet report, sending...
I20260925 10:08:29.581722 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.568639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35088 (local address 127.14.41.124:32911)
0925 10:08:29.568858 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.568861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.569940 (+ 1079us) server_negotiation.cc:407] Connection header received
0925 10:08:29.570039 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.570040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.570053 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.570094 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.570849 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.571562 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.572673 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.573001 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.573074 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.574881 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.574883 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.574883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.574891 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.577336 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.580317 (+ 2981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.580318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.580318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.580380 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.580640 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.580641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.580641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.580779 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.580855 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.581239 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.581328 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":53,"threads_started":1}
I20260925 10:08:29.581992 21206 ts_manager.cc:194] Registered new tserver with Master: bb35bd8e8a9f42b1aa31926ca80f711b (127.14.41.67:44021)
I20260925 10:08:29.582420 21846 heartbeater.cc:347] Connected to a master server at 127.14.41.126:43871
I20260925 10:08:29.582472 21846 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:29.582538 21846 heartbeater.cc:511] Master 127.14.41.126:43871 requested a full tablet report, sending...
I20260925 10:08:29.582939 20609 ts_manager.cc:194] Re-registered known tserver with Master: bb35bd8e8a9f42b1aa31926ca80f711b (127.14.41.67:44021)
I20260925 10:08:29.583315 20609 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:40163
I20260925 10:08:29.584122 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.568318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38100 (local address 127.14.41.126:43871)
0925 10:08:29.569232 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.569234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.569242 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:08:29.569262 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.569263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.569274 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.571982 (+ 2708us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.572020 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.573657 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.574881 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.575192 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.575480 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.577735 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.577737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.577737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.577745 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.582566 (+ 4821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.582998 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.582999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.583000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.583061 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.583269 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.583270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.583270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.583404 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.583841 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.583883 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.584003 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":88,"threads_started":1}
I20260925 10:08:29.585026 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.568626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59940 (local address 127.14.41.125:41713)
0925 10:08:29.572147 (+ 3521us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.572149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.572786 (+ 637us) server_negotiation.cc:407] Connection header received
0925 10:08:29.572809 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.572810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.572821 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.572905 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.573544 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.574281 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.575293 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.575588 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.575688 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.579206 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.579207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.579207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.579215 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.583533 (+ 4318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.583997 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.583998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.583998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.584053 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.584285 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.584285 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.584285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.584409 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.584493 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.584671 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.584912 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3501,"thread_start_us":46,"threads_started":1}
I20260925 10:08:29.586014 20609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:08:29.587523 20609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:08:29.588263 21168 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efc62ed2149648ee810e263591ecca7d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: NON_VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: true } } }
I20260925 10:08:29.588370 21168 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:29.589749 21863 raft_consensus.cc:1050] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d: attempting to promote NON_VOTER 7bd20a07dd664c12a0dcc7ddbab4e072 to VOTER
I20260925 10:08:29.589912 21863 consensus_queue.cc:237] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false } }
I20260925 10:08:29.590862 21217 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 1 FOLLOWER]: Refusing update from remote peer efc62ed2149648ee810e263591ecca7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 10:08:29.590873 21403 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 1 LEARNER]: Refusing update from remote peer efc62ed2149648ee810e263591ecca7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 10:08:29.591079 21357 consensus_queue.cc:1035] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 10:08:29.591190 21168 consensus_queue.cc:1035] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 10:08:29.591882 21169 raft_consensus.cc:2932] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 7bd20a07dd664c12a0dcc7ddbab4e072 (127.14.41.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false } } }
I20260925 10:08:29.592814 21863 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "efc62ed2149648ee810e263591ecca7d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false } } }
I20260925 10:08:29.592842 21169 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efc62ed2149648ee810e263591ecca7d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false } } }
I20260925 10:08:29.592902 21863 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:29.592926 21169 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:29.592967 21863 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efc62ed2149648ee810e263591ecca7d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false } } }
I20260925 10:08:29.593006 21863 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:29.593138 21217 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 7bd20a07dd664c12a0dcc7ddbab4e072 (127.14.41.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false } } }
I20260925 10:08:29.594458 21356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efc62ed2149648ee810e263591ecca7d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false } } }
I20260925 10:08:29.594550 21356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:08:29.594810 21403 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 7bd20a07dd664c12a0dcc7ddbab4e072 (127.14.41.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false } } }
I20260925 10:08:29.596760 21432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efc62ed2149648ee810e263591ecca7d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false } } }
I20260925 10:08:29.596860 21432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:08:29.597955 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.593384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:44021 (local address 127.0.0.1:46456)
0925 10:08:29.593686 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.593700 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.593744 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.594193 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.594195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.594197 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:29.594249 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.594251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.595526 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.595527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.596674 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.596677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.597731 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.597733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.597797 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.597801 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.597833 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.597878 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282}
I20260925 10:08:29.598382 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.593846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.14.41.67:44021)
0925 10:08:29.594004 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.594008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.594019 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:08:29.594062 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.594065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.594082 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.594176 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:29.594327 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.595432 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.596952 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.597629 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.598126 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.598220 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.598250 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.598292 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260925 10:08:29.599094 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.593551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:40287 (local address 127.0.0.1:35370)
0925 10:08:29.595402 (+ 1851us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.595410 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.595458 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.595715 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.595717 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.595720 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:29.595767 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.595769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.596812 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.596813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.597860 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.597863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.598856 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.598857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.598919 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.598925 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.598982 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.599025 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1839,"thread_start_us":90,"threads_started":1}
I20260925 10:08:29.599141 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.593656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.14.41.65:40287)
0925 10:08:29.593878 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.593883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.595480 (+ 1597us) server_negotiation.cc:407] Connection header received
0925 10:08:29.595532 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.595535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.595555 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.595639 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:29.595857 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.596729 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.597990 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.598752 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.598852 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.598926 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.599006 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.599055 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":115,"threads_started":1}
I20260925 10:08:29.599089 21779 tablet_service.cc:1467] Processing CreateTablet for tablet bc46be1206fe402788aa45034daa74df (DEFAULT_TABLE table=test-table [id=4c260ac001f64583a13e3eaea51020ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:29.599522 21779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc46be1206fe402788aa45034daa74df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:29.599720 21870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.593442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:44663 (local address 127.0.0.1:35406)
0925 10:08:29.595271 (+ 1829us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.595281 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.595332 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.595720 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.595721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.595723 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:29.595769 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.595782 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.597276 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.597277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.598336 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.598339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.599500 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.599501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.599565 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.599569 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.599612 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.599653 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1814,"mutex_wait_us":2,"thread_start_us":81,"threads_started":1}
I20260925 10:08:29.599758 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.594258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35406 (local address 127.14.41.66:44663)
0925 10:08:29.594456 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.594460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.595518 (+ 1058us) server_negotiation.cc:407] Connection header received
0925 10:08:29.595565 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.595567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.595583 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.595651 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:29.595862 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.597192 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.598421 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.599413 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.599499 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.599572 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.599638 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.599682 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260925 10:08:29.599781 21506 tablet_service.cc:1467] Processing CreateTablet for tablet bc46be1206fe402788aa45034daa74df (DEFAULT_TABLE table=test-table [id=4c260ac001f64583a13e3eaea51020ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:29.600174 21506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc46be1206fe402788aa45034daa74df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:29.600574 21643 tablet_service.cc:1467] Processing CreateTablet for tablet bc46be1206fe402788aa45034daa74df (DEFAULT_TABLE table=test-table [id=4c260ac001f64583a13e3eaea51020ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:29.600906 21643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc46be1206fe402788aa45034daa74df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:29.603183 21875 tablet_bootstrap.cc:492] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541: Bootstrap starting.
I20260925 10:08:29.603452 21876 tablet_bootstrap.cc:492] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b: Bootstrap starting.
I20260925 10:08:29.604103 21875 tablet_bootstrap.cc:654] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:29.604103 21876 tablet_bootstrap.cc:654] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:29.604419 21877 tablet_bootstrap.cc:492] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607: Bootstrap starting.
I20260925 10:08:29.604475 21876 log.cc:824] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:29.604679 21875 log.cc:824] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:29.605082 21877 tablet_bootstrap.cc:654] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:29.605320 21876 tablet_bootstrap.cc:492] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b: No bootstrap required, opened a new log
I20260925 10:08:29.605376 21876 ts_tablet_manager.cc:1397] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:08:29.605403 21877 log.cc:824] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:29.606144 21877 tablet_bootstrap.cc:492] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607: No bootstrap required, opened a new log
I20260925 10:08:29.606228 21877 ts_tablet_manager.cc:1397] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:08:29.607640 21876 raft_consensus.cc:348] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 } } peers { permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 } } peers { permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 } }
I20260925 10:08:29.607774 21876 raft_consensus.cc:374] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:29.607795 21876 raft_consensus.cc:729] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb35bd8e8a9f42b1aa31926ca80f711b, State: Initialized, Role: FOLLOWER
I20260925 10:08:29.607947 21876 consensus_queue.cc:260] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 } } peers { permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 } } peers { permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 } }
I20260925 10:08:29.608318 21876 ts_tablet_manager.cc:1428] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:08:29.608426 21877 raft_consensus.cc:348] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 } } peers { permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 } } peers { permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 } }
I20260925 10:08:29.608570 21875 tablet_bootstrap.cc:492] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541: No bootstrap required, opened a new log
I20260925 10:08:29.608588 21877 raft_consensus.cc:374] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:29.608632 21875 ts_tablet_manager.cc:1397] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260925 10:08:29.608642 21877 raft_consensus.cc:729] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 949ae6782e9d46b59871d4d1f6a05607, State: Initialized, Role: FOLLOWER
I20260925 10:08:29.608758 21877 consensus_queue.cc:260] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 } } peers { permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 } } peers { permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 } }
I20260925 10:08:29.608853 21846 heartbeater.cc:503] Master 127.14.41.126:43871 was elected leader, sending a full tablet report...
I20260925 10:08:29.609006 21877 ts_tablet_manager.cc:1428] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 10:08:29.609330 21710 heartbeater.cc:503] Master 127.14.41.126:43871 was elected leader, sending a full tablet report...
I20260925 10:08:29.611454 21875 raft_consensus.cc:348] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 } } peers { permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 } } peers { permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 } }
I20260925 10:08:29.611610 21875 raft_consensus.cc:374] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:29.611634 21875 raft_consensus.cc:729] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e02b6aac6c8f403396d2a6cfe768d541, State: Initialized, Role: FOLLOWER
I20260925 10:08:29.611773 21875 consensus_queue.cc:260] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 } } peers { permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 } } peers { permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 } }
I20260925 10:08:29.612061 21875 ts_tablet_manager.cc:1428] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260925 10:08:29.612124 21573 heartbeater.cc:503] Master 127.14.41.126:43871 was elected leader, sending a full tablet report...
W20260925 10:08:29.671883 21712 tablet.cc:2369] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:08:29.674600 21881 raft_consensus.cc:482] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:29.674729 21881 raft_consensus.cc:504] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 } } peers { permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 } } peers { permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 } }
I20260925 10:08:29.675140 21881 leader_election.cc:288] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 949ae6782e9d46b59871d4d1f6a05607 (127.14.41.66:44663), e02b6aac6c8f403396d2a6cfe768d541 (127.14.41.65:40287)
I20260925 10:08:29.678261 21849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.675350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:44663 (local address 127.14.41.67:59957)
0925 10:08:29.675495 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.675506 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.675553 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.675710 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.675711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.675712 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:29.675789 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.675790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.676485 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.676487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.677328 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.677331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.677936 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.677937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.678032 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.678035 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.678106 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.678152 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":140}
I20260925 10:08:29.678236 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.675477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:59957 (local address 127.14.41.66:44663)
0925 10:08:29.675532 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.675534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.675545 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:08:29.675633 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.675634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.675641 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.675707 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:29.675865 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.676384 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.677444 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.677846 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.677939 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.677949 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.678115 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.678156 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 10:08:29.678239 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.675476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:42981 (local address 127.14.41.65:40287)
0925 10:08:29.675523 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.675525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.675537 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:08:29.675581 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.675582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.675591 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.675664 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:29.675837 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.676401 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.677442 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.677853 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.677945 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.677954 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.678124 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.678165 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:08:29.678287 21851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.675350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:40287 (local address 127.14.41.67:42981)
0925 10:08:29.675426 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:29.675441 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:29.675513 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:29.675670 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:29.675671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:29.675673 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:29.675753 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.675754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.676504 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.676505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.677319 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:29.677329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.677942 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.677943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:29.678029 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.678033 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:08:29.678113 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 10:08:29.678159 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":34}
I20260925 10:08:29.678617 21663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc46be1206fe402788aa45034daa74df" candidate_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "949ae6782e9d46b59871d4d1f6a05607" is_pre_election: true
I20260925 10:08:29.678771 21526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc46be1206fe402788aa45034daa74df" candidate_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e02b6aac6c8f403396d2a6cfe768d541" is_pre_election: true
I20260925 10:08:29.678851 21663 raft_consensus.cc:2454] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb35bd8e8a9f42b1aa31926ca80f711b in term 0.
I20260925 10:08:29.678962 21526 raft_consensus.cc:2454] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb35bd8e8a9f42b1aa31926ca80f711b in term 0.
I20260925 10:08:29.679143 21735 leader_election.cc:302] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 949ae6782e9d46b59871d4d1f6a05607, bb35bd8e8a9f42b1aa31926ca80f711b; no voters:
I20260925 10:08:29.679293 21881 raft_consensus.cc:2781] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:29.679352 21881 raft_consensus.cc:482] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:29.679386 21881 raft_consensus.cc:3037] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:29.680282 21881 raft_consensus.cc:504] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 } } peers { permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 } } peers { permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 } }
I20260925 10:08:29.680482 21881 leader_election.cc:288] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [CANDIDATE]: Term 1 election: Requested vote from peers 949ae6782e9d46b59871d4d1f6a05607 (127.14.41.66:44663), e02b6aac6c8f403396d2a6cfe768d541 (127.14.41.65:40287)
I20260925 10:08:29.680601 21663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc46be1206fe402788aa45034daa74df" candidate_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "949ae6782e9d46b59871d4d1f6a05607"
I20260925 10:08:29.680701 21526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc46be1206fe402788aa45034daa74df" candidate_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e02b6aac6c8f403396d2a6cfe768d541"
I20260925 10:08:29.680717 21663 raft_consensus.cc:3037] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:29.680799 21526 raft_consensus.cc:3037] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:29.681677 21526 raft_consensus.cc:2454] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb35bd8e8a9f42b1aa31926ca80f711b in term 1.
I20260925 10:08:29.681677 21663 raft_consensus.cc:2454] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb35bd8e8a9f42b1aa31926ca80f711b in term 1.
I20260925 10:08:29.681921 21735 leader_election.cc:302] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 949ae6782e9d46b59871d4d1f6a05607, bb35bd8e8a9f42b1aa31926ca80f711b; no voters:
I20260925 10:08:29.682029 21881 raft_consensus.cc:2781] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:29.682183 21881 raft_consensus.cc:686] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [term 1 LEADER]: Becoming Leader. State: Replica: bb35bd8e8a9f42b1aa31926ca80f711b, State: Running, Role: LEADER
I20260925 10:08:29.682286 21881 consensus_queue.cc:237] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 } } peers { permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 } } peers { permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 } }
I20260925 10:08:29.683276 20608 catalog_manager.cc:5495] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b reported cstate change: term changed from 0 to 1, leader changed from <none> to bb35bd8e8a9f42b1aa31926ca80f711b (127.14.41.67). New cstate: current_term: 1 leader_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 } health_report { overall_health: UNKNOWN } } }
I20260925 10:08:29.694603 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.692126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.14.41.67:44021)
0925 10:08:29.692206 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.692208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.692218 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:08:29.692253 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.692254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.692273 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.692376 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:29.692540 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.693055 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.693806 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.694024 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.694083 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.694234 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:29.694350 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:29.694413 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.694455 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.694474 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:08:29.696724 21663 raft_consensus.cc:1261] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607 [term 1 FOLLOWER]: Refusing update from remote peer bb35bd8e8a9f42b1aa31926ca80f711b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:29.696724 21526 raft_consensus.cc:1261] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541 [term 1 FOLLOWER]: Refusing update from remote peer bb35bd8e8a9f42b1aa31926ca80f711b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:29.697142 21884 consensus_queue.cc:1035] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [LEADER]: Connected to new peer: Peer: permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:29.697299 21881 consensus_queue.cc:1035] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:29.700644 21889 mvcc.cc:205] Tried to move back new op lower bound from 7333195406114668544 to 7333195406059552768. Current Snapshot: MvccSnapshot[applied={T|T < 7333195406114668544}]
I20260925 10:08:29.700949 21888 mvcc.cc:205] Tried to move back new op lower bound from 7333195406114668544 to 7333195406059552768. Current Snapshot: MvccSnapshot[applied={T|T < 7333195406114668544}]
I20260925 10:08:29.701747 21887 mvcc.cc:205] Tried to move back new op lower bound from 7333195406114668544 to 7333195406059552768. Current Snapshot: MvccSnapshot[applied={T|T < 7333195406114668544}]
I20260925 10:08:29.703722 20608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:29.703815 20608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:29.705302 20608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P efc62ed2149648ee810e263591ecca7d: Sending DeleteTablet for 3 replicas of tablet bc46be1206fe402788aa45034daa74df
I20260925 10:08:29.705654 21779 tablet_service.cc:1514] Processing DeleteTablet for tablet bc46be1206fe402788aa45034daa74df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:29 UTC) from {username='oryx', principal='oryx/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:46456
I20260925 10:08:29.705714 21643 tablet_service.cc:1514] Processing DeleteTablet for tablet bc46be1206fe402788aa45034daa74df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:29 UTC) from {username='oryx', principal='oryx/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:35406
I20260925 10:08:29.705866 21897 tablet_replica.cc:330] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b: stopping tablet replica
I20260925 10:08:29.705966 21897 raft_consensus.cc:2229] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:08:29.705978 21898 tablet_replica.cc:330] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607: stopping tablet replica
I20260925 10:08:29.705966 21506 tablet_service.cc:1514] Processing DeleteTablet for tablet bc46be1206fe402788aa45034daa74df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:29 UTC) from {username='oryx', principal='oryx/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:35370
I20260925 10:08:29.706060 21898 raft_consensus.cc:2229] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:29.706089 21897 raft_consensus.cc:2258] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:29.706147 21898 raft_consensus.cc:2258] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:29.706655 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 20577
I20260925 10:08:29.706730 21898 ts_tablet_manager.cc:1905] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:29.708878 21898 ts_tablet_manager.cc:1918] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:08:29.708958 21898 log.cc:1196] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/bc46be1206fe402788aa45034daa74df
I20260925 10:08:29.709229 21898 ts_tablet_manager.cc:1939] T bc46be1206fe402788aa45034daa74df P 949ae6782e9d46b59871d4d1f6a05607: Deleting consensus metadata
I20260925 10:08:29.712558 21897 ts_tablet_manager.cc:1905] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 10:08:29.713850 21191 connection.cc:535] server connection from 127.0.0.1:59926 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 10:08:29.714951 21897 ts_tablet_manager.cc:1918] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:08:29.715022 21897 log.cc:1196] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/bc46be1206fe402788aa45034daa74df
I20260925 10:08:29.715317 21897 ts_tablet_manager.cc:1939] T bc46be1206fe402788aa45034daa74df P bb35bd8e8a9f42b1aa31926ca80f711b: Deleting consensus metadata
I20260925 10:08:29.716046 21899 tablet_replica.cc:330] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541: stopping tablet replica
I20260925 10:08:29.716179 21899 raft_consensus.cc:2229] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:29.716312 21899 raft_consensus.cc:2258] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 10:08:29.716573 21599 connection.cc:535] server connection from 127.0.0.1:35406 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 10:08:29.717303 21735 connection.cc:441] server connection from 127.0.0.1:46456 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46456 (request call id 1) could send its response
I20260925 10:08:29.718566 21899 ts_tablet_manager.cc:1905] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:29.720361 21899 ts_tablet_manager.cc:1918] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:08:29.720449 21899 log.cc:1196] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-0/wal/wals/bc46be1206fe402788aa45034daa74df
I20260925 10:08:29.720746 21899 ts_tablet_manager.cc:1939] T bc46be1206fe402788aa45034daa74df P e02b6aac6c8f403396d2a6cfe768d541: Deleting consensus metadata
W20260925 10:08:29.721278 21462 connection.cc:441] server connection from 127.0.0.1:35370 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35370 (request call id 1) could send its response
I20260925 10:08:29.726622 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.706737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.14.41.124:32911)
0925 10:08:29.707070 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.707072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.712261 (+ 5189us) server_negotiation.cc:407] Connection header received
0925 10:08:29.715535 (+ 3274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.715537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.715557 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.715594 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.717448 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.718199 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.722025 (+ 3826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.722236 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.722304 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.723899 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.723901 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.723901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.723910 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.725481 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.725903 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.725903 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.725904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.725953 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.726151 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.726151 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.726151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.726257 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.726329 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.726483 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.726554 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311}
I20260925 10:08:29.734321 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.727247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59956 (local address 127.14.41.125:41713)
0925 10:08:29.727299 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.727301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.727308 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:08:29.727332 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.727333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.727346 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.727400 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.728228 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.729006 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.729918 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.730136 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.730208 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.731500 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.731501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.731502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.731509 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.733086 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.733504 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.733505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.733506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.733566 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.733759 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.733761 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.733761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.733898 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.734010 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.734125 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.734218 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:08:29.734649 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:29.727393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.14.41.124:32911)
0925 10:08:29.727431 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:29.727433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:29.727476 (+ 43us) server_negotiation.cc:407] Connection header received
0925 10:08:29.727541 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:29.727542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:29.727551 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:29.727604 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:29.728287 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.728755 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.729741 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:29.730052 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:29.730125 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:29.731587 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:29.731588 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:29.731589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:29.731595 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:29.733341 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.733696 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.733697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.733697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.733747 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:29.734053 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:29.734054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:29.734055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:29.734184 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:29.734280 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:29.734475 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 10:08:29.734549 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:08:30.701988 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:30.701685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.67:39715)
0925 10:08:30.701823 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:30.701927 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
W20260925 10:08:30.702301 21846 heartbeater.cc:644] Failed to heartbeat to 127.14.41.126:43871 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
I20260925 10:08:30.702354 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:30.701714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.66:34199)
0925 10:08:30.702002 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:30.702296 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":74,"threads_started":1}
I20260925 10:08:30.702531 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:30.702099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.65:41855)
0925 10:08:30.702420 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:30.702489 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":46,"threads_started":1}
W20260925 10:08:30.702795 21710 heartbeater.cc:644] Failed to heartbeat to 127.14.41.126:43871 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
W20260925 10:08:30.702874 21573 heartbeater.cc:644] Failed to heartbeat to 127.14.41.126:43871 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
I20260925 10:08:31.206279 21910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader efc62ed2149648ee810e263591ecca7d)
I20260925 10:08:31.206303 21911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader efc62ed2149648ee810e263591ecca7d)
I20260925 10:08:31.206467 21911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false } }
I20260925 10:08:31.206488 21910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false } }
I20260925 10:08:31.206746 21910 leader_election.cc:288] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers efc62ed2149648ee810e263591ecca7d (127.14.41.126:43871), 86de7b8cfd2f42429708bdc3fa39e704 (127.14.41.125:41713)
I20260925 10:08:31.206887 21911 leader_election.cc:288] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers efc62ed2149648ee810e263591ecca7d (127.14.41.126:43871), 7bd20a07dd664c12a0dcc7ddbab4e072 (127.14.41.124:32911)
I20260925 10:08:31.207216 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:31.206921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:38568)
0925 10:08:31.207096 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:31.207168 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":46,"threads_started":1}
W20260925 10:08:31.207343 21377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
I20260925 10:08:31.207377 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:31.207023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:38584)
0925 10:08:31.207163 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:31.207214 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
W20260925 10:08:31.207617 21191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
I20260925 10:08:31.207795 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:31.207701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:38588)
0925 10:08:31.207739 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:31.207766 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 10:08:31.207890 21377 leader_election.cc:334] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer efc62ed2149648ee810e263591ecca7d (127.14.41.126:43871): Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
I20260925 10:08:31.208313 21916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:31.207911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:38592)
0925 10:08:31.208240 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:31.208282 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313}
W20260925 10:08:31.208465 21191 leader_election.cc:334] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer efc62ed2149648ee810e263591ecca7d (127.14.41.126:43871): Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
I20260925 10:08:31.210595 21914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:31.207087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:41713 (local address 127.0.0.1:59970)
0925 10:08:31.207165 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:31.207175 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:31.207539 (+ 364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:31.207780 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:31.207781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:31.207784 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:31.207847 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:31.207850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:31.208748 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:31.208749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:31.209612 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:31.209615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:31.210328 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:31.210330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:31.210388 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:31.210393 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:08:31.210479 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 10:08:31.210528 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":38,"threads_started":1}
I20260925 10:08:31.210647 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:31.207522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59970 (local address 127.14.41.125:41713)
0925 10:08:31.207606 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:31.207607 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:31.207614 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:08:31.207638 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:31.207638 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:31.207648 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:31.207715 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:31.208006 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:31.208670 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:31.209699 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:31.210234 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:31.210306 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:31.210316 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:31.210505 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 10:08:31.210550 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":35,"threads_started":1}
I20260925 10:08:31.210721 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:31.207391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:32911 (local address 127.0.0.1:35118)
0925 10:08:31.207434 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:31.207445 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:31.207505 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:31.208049 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:31.208050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:31.208054 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:31.208113 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:31.208114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:31.209041 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:31.209042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:31.209870 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:31.209872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:31.210507 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:31.210508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:31.210565 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:31.210568 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:31.210625 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 10:08:31.210662 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"thread_start_us":97,"threads_started":1}
I20260925 10:08:31.210892 21915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:31.207308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.14.41.124:32911)
0925 10:08:31.207750 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:31.207751 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:31.207759 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:08:31.207782 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:31.207783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:31.207793 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:31.208036 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:31.208195 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:31.208949 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:31.209967 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:31.210433 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:31.210503 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:31.210511 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:31.210643 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 10:08:31.210685 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":50,"threads_started":1}
I20260925 10:08:31.211138 21217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "86de7b8cfd2f42429708bdc3fa39e704" is_pre_election: true
I20260925 10:08:31.211138 21403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86de7b8cfd2f42429708bdc3fa39e704" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" is_pre_election: true
I20260925 10:08:31.211253 21403 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86de7b8cfd2f42429708bdc3fa39e704 in term 1.
I20260925 10:08:31.211253 21217 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bd20a07dd664c12a0dcc7ddbab4e072 in term 1.
I20260925 10:08:31.211452 21377 leader_election.cc:302] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bd20a07dd664c12a0dcc7ddbab4e072, 86de7b8cfd2f42429708bdc3fa39e704; no voters: efc62ed2149648ee810e263591ecca7d
I20260925 10:08:31.211449 21191 leader_election.cc:302] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7bd20a07dd664c12a0dcc7ddbab4e072, 86de7b8cfd2f42429708bdc3fa39e704; no voters: efc62ed2149648ee810e263591ecca7d
I20260925 10:08:31.211592 21911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 10:08:31.211584 21910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 10:08:31.211632 21910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 1 FOLLOWER]: Starting leader election (detected failure of leader efc62ed2149648ee810e263591ecca7d)
I20260925 10:08:31.211633 21911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 1 FOLLOWER]: Starting leader election (detected failure of leader efc62ed2149648ee810e263591ecca7d)
I20260925 10:08:31.211650 21910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:08:31.211658 21911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:08:31.212530 21911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false } }
I20260925 10:08:31.212592 21910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false } }
I20260925 10:08:31.212725 21910 leader_election.cc:288] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [CANDIDATE]: Term 2 election: Requested vote from peers efc62ed2149648ee810e263591ecca7d (127.14.41.126:43871), 86de7b8cfd2f42429708bdc3fa39e704 (127.14.41.125:41713)
I20260925 10:08:31.212721 21911 leader_election.cc:288] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [CANDIDATE]: Term 2 election: Requested vote from peers efc62ed2149648ee810e263591ecca7d (127.14.41.126:43871), 7bd20a07dd664c12a0dcc7ddbab4e072 (127.14.41.124:32911)
I20260925 10:08:31.212978 21914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:31.212849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:38604)
0925 10:08:31.212923 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:31.212946 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 10:08:31.212950 21217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "86de7b8cfd2f42429708bdc3fa39e704"
I20260925 10:08:31.212957 21403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86de7b8cfd2f42429708bdc3fa39e704" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072"
I20260925 10:08:31.213057 21217 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7bd20a07dd664c12a0dcc7ddbab4e072 in current term 2: Already voted for candidate 86de7b8cfd2f42429708bdc3fa39e704 in this term.
I20260925 10:08:31.213076 21403 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 86de7b8cfd2f42429708bdc3fa39e704 in current term 2: Already voted for candidate 7bd20a07dd664c12a0dcc7ddbab4e072 in this term.
I20260925 10:08:31.212911 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:31.212806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:38598)
0925 10:08:31.212848 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:31.212883 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 10:08:31.213457 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:31.213313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:38620)
0925 10:08:31.213362 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:31.213429 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 10:08:31.213521 21191 leader_election.cc:334] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer efc62ed2149648ee810e263591ecca7d (127.14.41.126:43871): Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
I20260925 10:08:31.213546 21191 leader_election.cc:302] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 86de7b8cfd2f42429708bdc3fa39e704; no voters: 7bd20a07dd664c12a0dcc7ddbab4e072, efc62ed2149648ee810e263591ecca7d
I20260925 10:08:31.213600 21914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:31.213423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:38634)
0925 10:08:31.213531 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:31.213567 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 10:08:31.213609 21911 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260925 10:08:31.213696 21377 leader_election.cc:334] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer efc62ed2149648ee810e263591ecca7d (127.14.41.126:43871): Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
I20260925 10:08:31.213743 21377 leader_election.cc:302] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7bd20a07dd664c12a0dcc7ddbab4e072; no voters: 86de7b8cfd2f42429708bdc3fa39e704, efc62ed2149648ee810e263591ecca7d
I20260925 10:08:31.213804 21910 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 10:08:31.703066 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:31.702804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.67:38799)
0925 10:08:31.702956 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:31.703005 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260925 10:08:31.703575 21921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:31.703260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.66:60017)
0925 10:08:31.703467 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:31.703504 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":65,"threads_started":1}
I20260925 10:08:31.703804 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:31.703484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.65:42271)
0925 10:08:31.703625 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:31.703770 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":69,"threads_started":1}
I20260925 10:08:32.704198 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:32.703947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.66:57209)
0925 10:08:32.704094 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:32.704140 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":54,"threads_started":1}
I20260925 10:08:32.704455 21925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:32.704197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.65:56207)
0925 10:08:32.704346 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:32.704401 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":57,"threads_started":1}
I20260925 10:08:32.704695 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:32.703799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.67:57373)
0925 10:08:32.704614 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:32.704653 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":71,"threads_started":1}
I20260925 10:08:32.871873 21926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:32.872001 21926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false } }
I20260925 10:08:32.872228 21926 leader_election.cc:288] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers efc62ed2149648ee810e263591ecca7d (127.14.41.126:43871), 7bd20a07dd664c12a0dcc7ddbab4e072 (127.14.41.124:32911)
I20260925 10:08:32.872622 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:32.872394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:60306)
0925 10:08:32.872525 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:32.872570 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260925 10:08:32.872732 21403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86de7b8cfd2f42429708bdc3fa39e704" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" is_pre_election: true
I20260925 10:08:32.872862 21403 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86de7b8cfd2f42429708bdc3fa39e704 in term 2.
I20260925 10:08:32.873132 21191 leader_election.cc:302] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bd20a07dd664c12a0dcc7ddbab4e072, 86de7b8cfd2f42429708bdc3fa39e704; no voters:
I20260925 10:08:32.873235 21926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 10:08:32.873235 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:32.873088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:60318)
0925 10:08:32.873150 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:32.873203 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 10:08:32.873288 21926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:32.873396 21926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 2 FOLLOWER]: Advancing to term 3
W20260925 10:08:32.873461 21191 leader_election.cc:334] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer efc62ed2149648ee810e263591ecca7d (127.14.41.126:43871): Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
I20260925 10:08:32.874158 21926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false } }
I20260925 10:08:32.874290 21926 leader_election.cc:288] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [CANDIDATE]: Term 3 election: Requested vote from peers efc62ed2149648ee810e263591ecca7d (127.14.41.126:43871), 7bd20a07dd664c12a0dcc7ddbab4e072 (127.14.41.124:32911)
I20260925 10:08:32.874469 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:32.874374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:60326)
0925 10:08:32.874412 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:32.874445 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 10:08:32.874570 21403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "86de7b8cfd2f42429708bdc3fa39e704" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072"
I20260925 10:08:32.874675 21403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 2 FOLLOWER]: Advancing to term 3
I20260925 10:08:32.875604 21403 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86de7b8cfd2f42429708bdc3fa39e704 in term 3.
I20260925 10:08:32.875837 21191 leader_election.cc:302] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bd20a07dd664c12a0dcc7ddbab4e072, 86de7b8cfd2f42429708bdc3fa39e704; no voters:
I20260925 10:08:32.875968 21926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 10:08:32.876091 21926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [term 3 LEADER]: Becoming Leader. State: Replica: 86de7b8cfd2f42429708bdc3fa39e704, State: Running, Role: LEADER
I20260925 10:08:32.876143 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:32.876034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:60338)
0925 10:08:32.876072 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:32.876106 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260925 10:08:32.876260 21191 leader_election.cc:334] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer efc62ed2149648ee810e263591ecca7d (127.14.41.126:43871): Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
I20260925 10:08:32.876220 21926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false } }
I20260925 10:08:32.876674 21929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86de7b8cfd2f42429708bdc3fa39e704. Latest consensus state: current_term: 3 leader_uuid: "86de7b8cfd2f42429708bdc3fa39e704" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false } } }
I20260925 10:08:32.876873 21929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:32.877115 21931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:32.877538 21931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:32.877599 21931 catalog_manager.cc:1259] Loaded cluster ID: 5b434835f51047d689e6ab46671765f6
I20260925 10:08:32.877615 21931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:08:32.877816 21931 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:32.877900 21931 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: Loaded TSK: 0
I20260925 10:08:32.878052 21931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:08:32.903163 21206 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:08:32.904958 21206 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:08:32.906250 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:32.906128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:60350)
0925 10:08:32.906188 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:32.906222 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 10:08:32.906236 21403 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [term 3 FOLLOWER]: Refusing update from remote peer 86de7b8cfd2f42429708bdc3fa39e704: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 10:08:32.906620 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:32.906510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:60362)
0925 10:08:32.906554 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:32.906579 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 10:08:32.906630 21929 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260925 10:08:32.906785 21191 consensus_peers.cc:487] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 -> Peer efc62ed2149648ee810e263591ecca7d (127.14.41.126:43871): Couldn't send request to peer efc62ed2149648ee810e263591ecca7d. Status: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 10:08:32.907393 21933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86de7b8cfd2f42429708bdc3fa39e704. Latest consensus state: current_term: 3 leader_uuid: "86de7b8cfd2f42429708bdc3fa39e704" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false } } }
I20260925 10:08:32.907541 21933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:08:32.908154 21929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "86de7b8cfd2f42429708bdc3fa39e704" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false } } }
I20260925 10:08:32.908252 21929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:32.908299 21934 mvcc.cc:205] Tried to move back new op lower bound from 7333195419261050880 to 7333195419141701632. Current Snapshot: MvccSnapshot[applied={T|T < 7333195419261050880}]
I20260925 10:08:32.908380 21933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "86de7b8cfd2f42429708bdc3fa39e704" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "efc62ed2149648ee810e263591ecca7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 43871 } } peers { permanent_uuid: "86de7b8cfd2f42429708bdc3fa39e704" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 41713 } attrs { promote: false } } peers { permanent_uuid: "7bd20a07dd664c12a0dcc7ddbab4e072" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 32911 } attrs { promote: false } } }
I20260925 10:08:32.908457 21933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd20a07dd664c12a0dcc7ddbab4e072 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:08:32.914011 21939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:32.910626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44866 (local address 127.14.41.67:44021)
0925 10:08:32.910900 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:32.910902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:32.910910 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:08:32.910937 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:32.910938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:32.910950 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:32.911018 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:32.911203 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:32.911987 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:32.913212 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:32.913649 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:32.913744 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:32.913760 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:32.913900 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 10:08:32.913941 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":51,"threads_started":1}
I20260925 10:08:32.914121 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:32.910609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:40287 (local address 127.0.0.1:53748)
0925 10:08:32.910734 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:32.910745 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:32.910799 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:32.910964 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:32.910965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:32.910966 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:32.911023 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:32.911024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:32.911797 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:32.911797 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:32.913035 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:32.913037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:32.913841 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:32.913842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:32.913924 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:32.913930 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:08:32.913961 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 10:08:32.914013 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":31,"thread_start_us":82,"threads_started":1}
I20260925 10:08:32.914011 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:32.910503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:44021 (local address 127.0.0.1:44866)
0925 10:08:32.910739 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:32.910749 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:32.910809 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:32.911067 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:32.911067 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:32.911070 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:32.911117 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:32.911118 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:32.912064 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:32.912065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:32.913142 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:32.913143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:32.913745 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:32.913746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:32.913808 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:32.913817 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 10:08:32.913891 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 10:08:32.913934 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260925 10:08:32.914230 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:32.910690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53748 (local address 127.14.41.65:40287)
0925 10:08:32.910819 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:32.910822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:32.910830 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:08:32.910859 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:32.910860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:32.910870 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:32.910964 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:32.911079 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:32.911722 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:32.913135 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:32.913743 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:32.914082 (+ 339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:32.914098 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:32.914118 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 10:08:32.914157 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":62,"threads_started":1}
I20260925 10:08:32.914258 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:32.910503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:44663 (local address 127.0.0.1:38392)
0925 10:08:32.911168 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:32.911177 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:32.911218 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:32.911448 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:32.911449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:32.911451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:32.911494 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:32.911495 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:32.912353 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:32.912354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:32.913253 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:32.913255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:32.914010 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:32.914012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:32.914069 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:32.914072 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:32.914101 (+ 29us) client_negotiation.cc:240] Negotiation successful
0925 10:08:32.914136 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":267,"thread_start_us":151,"threads_started":1}
I20260925 10:08:32.914486 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:32.910584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38392 (local address 127.14.41.66:44663)
0925 10:08:32.911308 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:32.911311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:32.911318 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:08:32.911342 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:32.911342 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:32.911352 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:32.911429 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:32.911571 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:32.912261 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:32.913337 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:32.913922 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:32.914356 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:32.914373 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:32.914397 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 10:08:32.914431 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":72,"threads_started":1}
I20260925 10:08:32.914626 21506 tablet_service.cc:1467] Processing CreateTablet for tablet a01d2cf1d3e940208f6d8feffe75487a (DEFAULT_TABLE table=test-table [id=9006bb8e54474dd9be0cc042e1d06c62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:32.914753 21506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01d2cf1d3e940208f6d8feffe75487a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:32.915086 21779 tablet_service.cc:1467] Processing CreateTablet for tablet a01d2cf1d3e940208f6d8feffe75487a (DEFAULT_TABLE table=test-table [id=9006bb8e54474dd9be0cc042e1d06c62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:32.915235 21779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01d2cf1d3e940208f6d8feffe75487a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:32.916232 21943 tablet_bootstrap.cc:492] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541: Bootstrap starting.
I20260925 10:08:32.916272 21643 tablet_service.cc:1467] Processing CreateTablet for tablet a01d2cf1d3e940208f6d8feffe75487a (DEFAULT_TABLE table=test-table [id=9006bb8e54474dd9be0cc042e1d06c62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:32.916388 21643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a01d2cf1d3e940208f6d8feffe75487a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:32.916617 21944 tablet_bootstrap.cc:492] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b: Bootstrap starting.
I20260925 10:08:32.916641 21943 tablet_bootstrap.cc:654] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:32.917045 21944 tablet_bootstrap.cc:654] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:32.917320 21943 tablet_bootstrap.cc:492] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541: No bootstrap required, opened a new log
I20260925 10:08:32.917408 21943 ts_tablet_manager.cc:1397] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:32.917550 21943 raft_consensus.cc:348] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 } } peers { permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 } } peers { permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 } }
I20260925 10:08:32.917634 21943 raft_consensus.cc:374] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:32.917649 21943 raft_consensus.cc:729] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e02b6aac6c8f403396d2a6cfe768d541, State: Initialized, Role: FOLLOWER
I20260925 10:08:32.917739 21944 tablet_bootstrap.cc:492] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b: No bootstrap required, opened a new log
I20260925 10:08:32.917739 21943 consensus_queue.cc:260] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 } } peers { permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 } } peers { permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 } }
I20260925 10:08:32.917824 21944 ts_tablet_manager.cc:1397] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:32.917938 21943 ts_tablet_manager.cc:1428] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:08:32.917968 21944 raft_consensus.cc:348] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 } } peers { permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 } } peers { permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 } }
I20260925 10:08:32.918097 21944 raft_consensus.cc:374] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:32.918140 21944 raft_consensus.cc:729] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb35bd8e8a9f42b1aa31926ca80f711b, State: Initialized, Role: FOLLOWER
I20260925 10:08:32.918109 21948 tablet_bootstrap.cc:492] T a01d2cf1d3e940208f6d8feffe75487a P 949ae6782e9d46b59871d4d1f6a05607: Bootstrap starting.
I20260925 10:08:32.918197 21944 consensus_queue.cc:260] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 } } peers { permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 } } peers { permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 } }
I20260925 10:08:32.918305 21925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:32.918203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.65:58953)
0925 10:08:32.918241 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:32.918273 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 10:08:32.918427 21944 ts_tablet_manager.cc:1428] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 10:08:32.918840 21948 tablet_bootstrap.cc:654] T a01d2cf1d3e940208f6d8feffe75487a P 949ae6782e9d46b59871d4d1f6a05607: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:32.919091 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:32.918993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.67:33673)
0925 10:08:32.919034 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:32.919062 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 10:08:32.919504 21948 tablet_bootstrap.cc:492] T a01d2cf1d3e940208f6d8feffe75487a P 949ae6782e9d46b59871d4d1f6a05607: No bootstrap required, opened a new log
I20260925 10:08:32.919533 21574 heartbeater.cc:503] Master 127.14.41.125:41713 was elected leader, sending a full tablet report...
I20260925 10:08:32.919556 21948 ts_tablet_manager.cc:1397] T a01d2cf1d3e940208f6d8feffe75487a P 949ae6782e9d46b59871d4d1f6a05607: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:32.919781 21925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:32.919684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.65:44717)
0925 10:08:32.919720 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:32.919753 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 10:08:32.919780 21847 heartbeater.cc:503] Master 127.14.41.125:41713 was elected leader, sending a full tablet report...
I20260925 10:08:32.919656 21948 raft_consensus.cc:348] T a01d2cf1d3e940208f6d8feffe75487a P 949ae6782e9d46b59871d4d1f6a05607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 } } peers { permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 } } peers { permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 } }
I20260925 10:08:32.919902 21948 raft_consensus.cc:374] T a01d2cf1d3e940208f6d8feffe75487a P 949ae6782e9d46b59871d4d1f6a05607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:32.919955 21948 raft_consensus.cc:729] T a01d2cf1d3e940208f6d8feffe75487a P 949ae6782e9d46b59871d4d1f6a05607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 949ae6782e9d46b59871d4d1f6a05607, State: Initialized, Role: FOLLOWER
I20260925 10:08:32.920020 21948 consensus_queue.cc:260] T a01d2cf1d3e940208f6d8feffe75487a P 949ae6782e9d46b59871d4d1f6a05607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 } } peers { permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 } } peers { permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 } }
I20260925 10:08:32.920193 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:32.919842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.67:35549)
0925 10:08:32.920134 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:32.920165 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272}
I20260925 10:08:32.920200 21948 ts_tablet_manager.cc:1428] T a01d2cf1d3e940208f6d8feffe75487a P 949ae6782e9d46b59871d4d1f6a05607: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:32.920749 21711 heartbeater.cc:503] Master 127.14.41.125:41713 was elected leader, sending a full tablet report...
I20260925 10:08:32.920749 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:32.920629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.66:55473)
0925 10:08:32.920689 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:32.920719 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 10:08:33.037245 21575 tablet.cc:2369] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:08:33.068054 21848 tablet.cc:2369] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:08:33.265337 21947 raft_consensus.cc:482] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:33.265441 21947 raft_consensus.cc:504] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 } } peers { permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 } } peers { permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 } }
I20260925 10:08:33.265851 21947 leader_election.cc:288] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb35bd8e8a9f42b1aa31926ca80f711b (127.14.41.67:44021), 949ae6782e9d46b59871d4d1f6a05607 (127.14.41.66:44663)
I20260925 10:08:33.270138 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:33.266154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:34615 (local address 127.14.41.66:44663)
0925 10:08:33.266233 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:33.266236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:33.266245 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:08:33.266302 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:33.266303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:33.266315 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:33.266382 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:33.266605 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:33.267357 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:33.269042 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:33.269621 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:33.269711 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:33.269720 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:33.270008 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 10:08:33.270063 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:08:33.270136 21939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:33.266187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:51121 (local address 127.14.41.67:44021)
0925 10:08:33.266233 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:33.266236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:33.266324 (+ 88us) server_negotiation.cc:407] Connection header received
0925 10:08:33.266351 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:33.266352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:33.266362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:33.266414 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:33.266605 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:33.267396 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:33.269046 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:33.269621 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:33.269709 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:33.269720 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:33.270008 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 10:08:33.270063 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:08:33.270144 21925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:33.266027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:44663 (local address 127.14.41.65:34615)
0925 10:08:33.266095 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:33.266107 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:33.266176 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:33.266417 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:33.266418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:33.266422 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:33.266523 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:33.266525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:33.267509 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:33.267510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:33.268934 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:33.268936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:33.269772 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:33.269775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:33.269873 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:33.269880 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:08:33.269962 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 10:08:33.270032 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":101}
I20260925 10:08:33.270154 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:33.266027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:44021 (local address 127.14.41.65:51121)
0925 10:08:33.266234 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:33.266243 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:33.266299 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:33.266431 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:33.266432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:33.266433 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:33.266523 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:33.266525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:33.267509 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:33.267510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:33.268923 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:33.268927 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:33.269799 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:33.269800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:33.269886 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:33.269888 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:08:33.269962 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 10:08:33.270025 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":82,"thread_start_us":125,"threads_started":1}
I20260925 10:08:33.270576 21663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01d2cf1d3e940208f6d8feffe75487a" candidate_uuid: "e02b6aac6c8f403396d2a6cfe768d541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "949ae6782e9d46b59871d4d1f6a05607" is_pre_election: true
I20260925 10:08:33.270701 21663 raft_consensus.cc:2454] T a01d2cf1d3e940208f6d8feffe75487a P 949ae6782e9d46b59871d4d1f6a05607 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e02b6aac6c8f403396d2a6cfe768d541 in term 0.
I20260925 10:08:33.270874 21799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01d2cf1d3e940208f6d8feffe75487a" candidate_uuid: "e02b6aac6c8f403396d2a6cfe768d541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" is_pre_election: true
I20260925 10:08:33.271032 21799 raft_consensus.cc:2454] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e02b6aac6c8f403396d2a6cfe768d541 in term 0.
I20260925 10:08:33.271070 21462 leader_election.cc:302] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 949ae6782e9d46b59871d4d1f6a05607, e02b6aac6c8f403396d2a6cfe768d541; no voters:
I20260925 10:08:33.271270 21947 raft_consensus.cc:2781] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:33.271304 21947 raft_consensus.cc:482] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:33.271318 21947 raft_consensus.cc:3037] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:33.272056 21947 raft_consensus.cc:504] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 } } peers { permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 } } peers { permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 } }
I20260925 10:08:33.272203 21947 leader_election.cc:288] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [CANDIDATE]: Term 1 election: Requested vote from peers bb35bd8e8a9f42b1aa31926ca80f711b (127.14.41.67:44021), 949ae6782e9d46b59871d4d1f6a05607 (127.14.41.66:44663)
I20260925 10:08:33.272398 21799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01d2cf1d3e940208f6d8feffe75487a" candidate_uuid: "e02b6aac6c8f403396d2a6cfe768d541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b"
I20260925 10:08:33.272486 21799 raft_consensus.cc:3037] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:33.272415 21663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a01d2cf1d3e940208f6d8feffe75487a" candidate_uuid: "e02b6aac6c8f403396d2a6cfe768d541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "949ae6782e9d46b59871d4d1f6a05607"
I20260925 10:08:33.272529 21663 raft_consensus.cc:3037] T a01d2cf1d3e940208f6d8feffe75487a P 949ae6782e9d46b59871d4d1f6a05607 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:33.273187 21799 raft_consensus.cc:2454] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e02b6aac6c8f403396d2a6cfe768d541 in term 1.
I20260925 10:08:33.273375 21460 leader_election.cc:302] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb35bd8e8a9f42b1aa31926ca80f711b, e02b6aac6c8f403396d2a6cfe768d541; no voters:
I20260925 10:08:33.273478 21947 raft_consensus.cc:2781] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:33.273568 21663 raft_consensus.cc:2454] T a01d2cf1d3e940208f6d8feffe75487a P 949ae6782e9d46b59871d4d1f6a05607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e02b6aac6c8f403396d2a6cfe768d541 in term 1.
I20260925 10:08:33.273638 21947 raft_consensus.cc:686] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [term 1 LEADER]: Becoming Leader. State: Replica: e02b6aac6c8f403396d2a6cfe768d541, State: Running, Role: LEADER
I20260925 10:08:33.273718 21947 consensus_queue.cc:237] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 } } peers { permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 } } peers { permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 } }
I20260925 10:08:33.274101 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:33.273989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.65:33689)
0925 10:08:33.274027 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:33.274068 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 10:08:33.274670 21206 catalog_manager.cc:5495] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 reported cstate change: term changed from 0 to 1, leader changed from <none> to e02b6aac6c8f403396d2a6cfe768d541 (127.14.41.65). New cstate: current_term: 1 leader_uuid: "e02b6aac6c8f403396d2a6cfe768d541" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e02b6aac6c8f403396d2a6cfe768d541" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 } health_report { overall_health: UNKNOWN } } }
I20260925 10:08:33.357290 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:33.357125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:60366)
0925 10:08:33.357186 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:33.357238 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 10:08:33.357360 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:33.354866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53764 (local address 127.14.41.65:40287)
0925 10:08:33.354923 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:33.354925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:33.354941 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:08:33.354976 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:33.354977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:33.354995 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:33.355058 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:33.355173 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:33.355734 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:33.356556 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:33.356745 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:33.356813 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:33.356929 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:33.357089 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:33.357168 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:33.357238 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 10:08:33.357259 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 10:08:33.357729 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:33.357628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.0.0.1:60374)
0925 10:08:33.357679 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:33.357710 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 10:08:33.358505 21663 raft_consensus.cc:1261] T a01d2cf1d3e940208f6d8feffe75487a P 949ae6782e9d46b59871d4d1f6a05607 [term 1 FOLLOWER]: Refusing update from remote peer e02b6aac6c8f403396d2a6cfe768d541: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:33.358541 21799 raft_consensus.cc:1261] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b [term 1 FOLLOWER]: Refusing update from remote peer e02b6aac6c8f403396d2a6cfe768d541: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:33.358819 21953 consensus_queue.cc:1035] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "949ae6782e9d46b59871d4d1f6a05607" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 44663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:33.359006 21947 consensus_queue.cc:1035] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb35bd8e8a9f42b1aa31926ca80f711b" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:33.359802 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:33.359691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.66:41095)
0925 10:08:33.359738 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:33.359766 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 10:08:33.360121 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:33.360004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.67:35507)
0925 10:08:33.360051 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:33.360086 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 10:08:33.361294 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:33.361168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.65:53259)
0925 10:08:33.361230 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:33.361259 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 10:08:33.361733 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:33.361590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.67:45067)
0925 10:08:33.361668 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:33.361694 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 10:08:33.362257 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:33.362112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:43871 (local address 127.14.41.66:50227)
0925 10:08:33.362199 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:33.362224 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:43871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 10:08:33.363737 21207 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:33.363852 21207 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:33.365254 21207 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86de7b8cfd2f42429708bdc3fa39e704: Sending DeleteTablet for 3 replicas of tablet a01d2cf1d3e940208f6d8feffe75487a
I20260925 10:08:33.365553 21643 tablet_service.cc:1514] Processing DeleteTablet for tablet a01d2cf1d3e940208f6d8feffe75487a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:33 UTC) from {username='oryx', principal='oryx/127.14.41.125@KRBTEST.COM'} at 127.0.0.1:38392
I20260925 10:08:33.365566 21506 tablet_service.cc:1514] Processing DeleteTablet for tablet a01d2cf1d3e940208f6d8feffe75487a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:33 UTC) from {username='oryx', principal='oryx/127.14.41.125@KRBTEST.COM'} at 127.0.0.1:53748
I20260925 10:08:33.365778 21967 tablet_replica.cc:330] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541: stopping tablet replica
I20260925 10:08:33.365841 21967 raft_consensus.cc:2229] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:08:33.365931 21967 raft_consensus.cc:2258] T a01d2cf1d3e940208f6d8feffe75487a P e02b6aac6c8f403396d2a6cfe768d541 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:33.366073 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 21354
I20260925 10:08:33.366618 21779 tablet_service.cc:1514] Processing DeleteTablet for tablet a01d2cf1d3e940208f6d8feffe75487a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:33 UTC) from {username='oryx', principal='oryx/127.14.41.125@KRBTEST.COM'} at 127.0.0.1:44866
I20260925 10:08:33.366848 21968 tablet_replica.cc:330] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b: stopping tablet replica
I20260925 10:08:33.366910 21968 raft_consensus.cc:2229] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:33.366959 21968 raft_consensus.cc:2258] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:33.367393 21968 ts_tablet_manager.cc:1905] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:33.368374 21966 tablet_replica.cc:330] T a01d2cf1d3e940208f6d8feffe75487a P 949ae6782e9d46b59871d4d1f6a05607: stopping tablet replica
I20260925 10:08:33.368530 21966 raft_consensus.cc:2229] T a01d2cf1d3e940208f6d8feffe75487a P 949ae6782e9d46b59871d4d1f6a05607 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:33.368593 21966 raft_consensus.cc:2258] T a01d2cf1d3e940208f6d8feffe75487a P 949ae6782e9d46b59871d4d1f6a05607 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:33.369158 21966 ts_tablet_manager.cc:1905] T a01d2cf1d3e940208f6d8feffe75487a P 949ae6782e9d46b59871d4d1f6a05607: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:33.369448 21968 ts_tablet_manager.cc:1918] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:08:33.369514 21968 log.cc:1196] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/a01d2cf1d3e940208f6d8feffe75487a
I20260925 10:08:33.369756 21968 ts_tablet_manager.cc:1939] T a01d2cf1d3e940208f6d8feffe75487a P bb35bd8e8a9f42b1aa31926ca80f711b: Deleting consensus metadata
I20260925 10:08:33.370383 21192 catalog_manager.cc:4841] TS bb35bd8e8a9f42b1aa31926ca80f711b (127.14.41.67:44021): tablet a01d2cf1d3e940208f6d8feffe75487a (table test-table [id=9006bb8e54474dd9be0cc042e1d06c62]) successfully deleted
I20260925 10:08:33.370688 21966 ts_tablet_manager.cc:1918] T a01d2cf1d3e940208f6d8feffe75487a P 949ae6782e9d46b59871d4d1f6a05607: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:08:33.370764 21966 log.cc:1196] T a01d2cf1d3e940208f6d8feffe75487a P 949ae6782e9d46b59871d4d1f6a05607: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/a01d2cf1d3e940208f6d8feffe75487a
I20260925 10:08:33.371004 21966 ts_tablet_manager.cc:1939] T a01d2cf1d3e940208f6d8feffe75487a P 949ae6782e9d46b59871d4d1f6a05607: Deleting consensus metadata
I20260925 10:08:33.371547 21194 catalog_manager.cc:4841] TS 949ae6782e9d46b59871d4d1f6a05607 (127.14.41.66:44663): tablet a01d2cf1d3e940208f6d8feffe75487a (table test-table [id=9006bb8e54474dd9be0cc042e1d06c62]) successfully deleted
W20260925 10:08:33.373534 21940 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:33.373396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:40287 (local address 127.0.0.1:53766)
0925 10:08:33.373470 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:33.373505 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.65:40287: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 10:08:33.373739 21192 catalog_manager.cc:4584] TS e02b6aac6c8f403396d2a6cfe768d541 (127.14.41.65:40287): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a01d2cf1d3e940208f6d8feffe75487a: Network error: Client connection negotiation failed: client connection to 127.14.41.65:40287: connect: Connection reset by peer (error 104)
I20260925 10:08:33.373804 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 21580
I20260925 10:08:33.379673 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 21716
I20260925 10:08:33.385656 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 21073
I20260925 10:08:33.390849 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 21261
2026-09-25T10:08:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6645 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:08:33 dist-test-slave-s6tm krb5kdc[21971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:08:33 dist-test-slave-s6tm krb5kdc[21971](info): set up 2 sockets
Sep 25 10:08:33 dist-test-slave-s6tm krb5kdc[21971](info): commencing operation
krb5kdc: starting...
W20260925 10:08:34.450006 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:08:34 dist-test-slave-s6tm krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:08:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:34Z Disabled control of system clock
WARNING: no policy specified for oryx/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.126.keytab.
Entry for principal oryx/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.126.keytab.
I20260925 10:08:34.578696 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:34013
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.126.keytab
--principal=oryx/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:34013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:34.662894 21984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:34.663154 21984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:34.663192 21984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:34.664393 21984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:34.664469 21984 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 10:08:34.664490 21984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:34.664513 21984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:34.664546 21984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:34.664567 21984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:08:34.666111 21984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:34013
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.126.keytab
--principal=oryx/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:34013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:34.666399 21984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:34.666627 21984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:34.671878 21989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:34.671906 21992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:34.671906 21990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:34.672211 21984 server_base.cc:1034] running on GCE node
I20260925 10:08:34.672478 21984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:34.672812 21984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:34.673972 21984 hybrid_clock.cc:648] HybridClock initialized: now 1790330914673943 us; error 50 us; skew 500 ppm
Sep 25 10:08:34 dist-test-slave-s6tm krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330914, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:34.676971 21984 init.cc:377] Logged in from keytab as oryx/127.14.41.126@KRBTEST.COM (short username oryx)
I20260925 10:08:34.677574 21984 webserver.cc:460] Webserver started at http://127.14.41.126:42787/ using document root <none> and password file <none>
I20260925 10:08:34.677824 21984 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:34.677870 21984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:34.678048 21984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:34.679193 21984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "4bd1639b766e409eac6dcf2e872f66de"
format_stamp: "Formatted at 2026-09-25 10:08:34 on dist-test-slave-s6tm"
server_key: "5e359ce0834bc89731ea788d0317ccee"
server_key_iv: "a6fbc09078c13eb129e0847d14867b32"
server_key_version: "encryptionkey@0"
I20260925 10:08:34.679597 21984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "4bd1639b766e409eac6dcf2e872f66de"
format_stamp: "Formatted at 2026-09-25 10:08:34 on dist-test-slave-s6tm"
server_key: "5e359ce0834bc89731ea788d0317ccee"
server_key_iv: "a6fbc09078c13eb129e0847d14867b32"
server_key_version: "encryptionkey@0"
I20260925 10:08:34.681159 21984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:08:34.682114 21999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:34.682353 21984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:34.682456 21984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "4bd1639b766e409eac6dcf2e872f66de"
format_stamp: "Formatted at 2026-09-25 10:08:34 on dist-test-slave-s6tm"
server_key: "5e359ce0834bc89731ea788d0317ccee"
server_key_iv: "a6fbc09078c13eb129e0847d14867b32"
server_key_version: "encryptionkey@0"
I20260925 10:08:34.682515 21984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:34.706348 21984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:34.708937 21984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:34.709060 21984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:34.713850 21984 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:34013
I20260925 10:08:34.713989 22061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:34013 every 8 connection(s)
I20260925 10:08:34.714313 21984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:34.715036 22062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:34.717433 22062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de: Bootstrap starting.
I20260925 10:08:34.718068 22062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:34.718391 22062 log.cc:824] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:34.719188 22062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de: No bootstrap required, opened a new log
I20260925 10:08:34.721383 22062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd1639b766e409eac6dcf2e872f66de" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 34013 } }
I20260925 10:08:34.721567 22062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:34.721593 22062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd1639b766e409eac6dcf2e872f66de, State: Initialized, Role: FOLLOWER
I20260925 10:08:34.721771 22062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd1639b766e409eac6dcf2e872f66de" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 34013 } }
I20260925 10:08:34.721855 22062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:08:34.721879 22062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:34.721951 22062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:34.722553 22062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd1639b766e409eac6dcf2e872f66de" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 34013 } }
I20260925 10:08:34.722649 22062 leader_election.cc:302] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bd1639b766e409eac6dcf2e872f66de; no voters:
I20260925 10:08:34.722877 22062 leader_election.cc:288] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:08:34.722978 22065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:34.723140 22062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:34.723141 22065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de [term 1 LEADER]: Becoming Leader. State: Replica: 4bd1639b766e409eac6dcf2e872f66de, State: Running, Role: LEADER
I20260925 10:08:34.723292 22065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd1639b766e409eac6dcf2e872f66de" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 34013 } }
I20260925 10:08:34.723690 22067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bd1639b766e409eac6dcf2e872f66de. Latest consensus state: current_term: 1 leader_uuid: "4bd1639b766e409eac6dcf2e872f66de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd1639b766e409eac6dcf2e872f66de" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 34013 } } }
I20260925 10:08:34.723686 22066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bd1639b766e409eac6dcf2e872f66de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd1639b766e409eac6dcf2e872f66de" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 34013 } } }
I20260925 10:08:34.723814 22067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:34.723819 22066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:34.724308 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 21984
I20260925 10:08:34.724455 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:08:34.724702 14501 external_mini_cluster.cc:1428] Setting key 741fb6caa961e2bd1bc052a7293de6c4
W20260925 10:08:34.725209 22081 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:08:34.725265 22081 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:08:34.725322 22074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:34.725492 22074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:34.727239 22074 catalog_manager.cc:1347] Generated new cluster ID: 0c3c09e52d904ae494373d213b48fe90
I20260925 10:08:34.727300 22074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:08:34 dist-test-slave-s6tm krb5kdc[21971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM
I20260925 10:08:34.738096 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:34.725275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43830 (local address 127.14.41.126:34013)
0925 10:08:34.725439 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:34.725449 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:34.726315 (+ 866us) server_negotiation.cc:407] Connection header received
0925 10:08:34.726490 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:34.726497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:34.726544 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:34.726690 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:34.727555 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:34.729032 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:34.730127 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:34.730783 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:34.731059 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:34.734162 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:34.734168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:34.734168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:34.734181 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:34.736191 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:34.736584 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:34.736587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:34.736588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:34.736749 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:34.736958 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:34.736958 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:34.736959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:34.737297 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:34.737436 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:34.737708 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 10:08:34.737861 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260925 10:08:34.738180 22074 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:08:34.738886 22074 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:34.749589 22074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de: Generated new TSK 0
I20260925 10:08:34.749931 22074 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.65.keytab.
Entry for principal oryx/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.65.keytab.
I20260925 10:08:34.789412 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:34013
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.65.keytab
--principal=oryx/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:34.889807 22088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:34.890030 22088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:34.890056 22088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:34.891235 22088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:34.891283 22088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:34.891304 22088 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:08:34.891348 22088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:08:34.893002 22088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.65.keytab
--principal=oryx/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.41.126:34013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:34.893261 22088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:34.893550 22088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:08:34.899034 22088 server_base.cc:1034] running on GCE node
W20260925 10:08:34.898990 22093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:34.899031 22094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:34.898990 22096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:34.899434 22088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:34.899720 22088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:34.900897 22088 hybrid_clock.cc:648] HybridClock initialized: now 1790330914900868 us; error 61 us; skew 500 ppm
Sep 25 10:08:34 dist-test-slave-s6tm krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330914, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:34.907526 22088 init.cc:377] Logged in from keytab as oryx/127.14.41.65@KRBTEST.COM (short username oryx)
I20260925 10:08:34.908057 22088 webserver.cc:460] Webserver started at http://127.14.41.65:35795/ using document root <none> and password file <none>
I20260925 10:08:34.908310 22088 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:34.908380 22088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:34.908538 22088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:34.909631 22088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "79d20d5f21224c2091e691e77fce586f"
format_stamp: "Formatted at 2026-09-25 10:08:34 on dist-test-slave-s6tm"
server_key: "97c2470f086efcd1c9a94502d4769a20"
server_key_iv: "148376f5617e82ff333cb21249790a49"
server_key_version: "encryptionkey@0"
I20260925 10:08:34.910015 22088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "79d20d5f21224c2091e691e77fce586f"
format_stamp: "Formatted at 2026-09-25 10:08:34 on dist-test-slave-s6tm"
server_key: "97c2470f086efcd1c9a94502d4769a20"
server_key_iv: "148376f5617e82ff333cb21249790a49"
server_key_version: "encryptionkey@0"
I20260925 10:08:34.911352 22088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:08:34.912215 22103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:34.912443 22088 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:34.912540 22088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "79d20d5f21224c2091e691e77fce586f"
format_stamp: "Formatted at 2026-09-25 10:08:34 on dist-test-slave-s6tm"
server_key: "97c2470f086efcd1c9a94502d4769a20"
server_key_iv: "148376f5617e82ff333cb21249790a49"
server_key_version: "encryptionkey@0"
I20260925 10:08:34.912623 22088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:34.922000 22088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:34.924566 22088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:34.924865 22088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:34.925522 22088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:34.925582 22088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:34.925626 22088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:34.925660 22088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:34.932832 22088 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:42479
I20260925 10:08:34.932932 22218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:42479 every 8 connection(s)
Sep 25 10:08:34 dist-test-slave-s6tm krb5kdc[21971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330914, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM
I20260925 10:08:34.933259 22088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:08:34.935683 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 22088
I20260925 10:08:34.935815 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:08:34.936162 14501 external_mini_cluster.cc:1428] Setting key bde86d252244d6fbe3836f28fe5cb00a
I20260925 10:08:34.938868 22112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:34.925961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:34013 (local address 127.14.41.65:35101)
0925 10:08:34.926120 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:34.926155 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:34.926669 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:34.927004 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:34.927009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:34.927445 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:34.927769 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:34.927779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:34.928841 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:34.928842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:34.930362 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:34.930366 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:34.930860 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:34.930861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:34.931028 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:34.931716 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:34.931727 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:34.933767 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:34.936857 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:34.936867 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:34.936869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:34.937367 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:34.937618 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:34.937619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:34.937620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:34.937765 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:34.938088 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:34.938094 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:34.938268 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 10:08:34.938376 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:08:34.938571 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260925 10:08:34.939230 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:34.925970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:35101 (local address 127.14.41.126:34013)
0925 10:08:34.926040 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:34.926042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:34.926235 (+ 193us) server_negotiation.cc:407] Connection header received
0925 10:08:34.926787 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:34.926789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:34.926806 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:34.926876 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:34.927885 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:34.928734 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:34.930460 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:34.930768 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:34.930828 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:34.933880 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:34.933883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:34.933883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:34.933892 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:34.936759 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:34.937492 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:34.937493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:34.937494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:34.937555 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:34.937847 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:34.937848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:34.937849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:34.937997 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:34.938062 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:34.939051 (+ 989us) server_negotiation.cc:299] Negotiation successful
0925 10:08:34.939141 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:08:34.943017 22219 heartbeater.cc:347] Connected to a master server at 127.14.41.126:34013
I20260925 10:08:34.943145 22219 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:34.943395 22219 heartbeater.cc:511] Master 127.14.41.126:34013 requested a full tablet report, sending...
I20260925 10:08:34.945019 22015 ts_manager.cc:194] Registered new tserver with Master: 79d20d5f21224c2091e691e77fce586f (127.14.41.65:42479)
I20260925 10:08:34.946044 22015 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:35101
WARNING: no policy specified for oryx/127.14.41.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.41.66@KRBTEST.COM" created.
I20260925 10:08:34.957577 22226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:34.944874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:34013 (local address 127.14.41.65:52555)
0925 10:08:34.945440 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:34.945453 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:34.945885 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:34.946852 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:34.946854 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:34.947114 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:34.948704 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:34.948708 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:34.949755 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:34.949757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:34.950683 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:34.950686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:34.951186 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:34.951188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:34.951270 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:34.951852 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:34.951858 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:34.952444 (+ 586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:34.955737 (+ 3293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:34.955738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:34.955739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:34.956046 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:34.956713 (+ 667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:34.956714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:34.956714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:34.956795 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:34.957164 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:34.957165 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:34.957229 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 10:08:34.957298 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 10:08:34.957442 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":78,"threads_started":1}
I20260925 10:08:34.957619 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:34.946309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:52555 (local address 127.14.41.126:34013)
0925 10:08:34.946358 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:34.946360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:34.946368 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:08:34.946394 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:34.946395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:34.946407 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:34.946474 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:34.948830 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:34.949640 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:34.950765 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:34.951101 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:34.951173 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:34.952596 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:34.952598 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:34.952598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:34.952606 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:34.955641 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:34.956563 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:34.956564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:34.956564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:34.956629 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:34.956897 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:34.956898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:34.956899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:34.957078 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:34.957164 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:34.957344 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 10:08:34.957458 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.66.keytab.
Entry for principal oryx/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.66.keytab.
I20260925 10:08:34.975837 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:34013
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.66.keytab
--principal=oryx/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:35.067200 22228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:35.067433 22228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:35.067461 22228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:35.068742 22228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:35.068787 22228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:35.068809 22228 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:08:35.068828 22228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:08:35.070425 22228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.66.keytab
--principal=oryx/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.41.126:34013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:35.070711 22228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:35.070955 22228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:35.076383 22234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:35.076370 22236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:35.076354 22233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:35.076409 22228 server_base.cc:1034] running on GCE node
I20260925 10:08:35.076869 22228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:35.077147 22228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:35.078352 22228 hybrid_clock.cc:648] HybridClock initialized: now 1790330915078315 us; error 40 us; skew 500 ppm
Sep 25 10:08:35 dist-test-slave-s6tm krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330915, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:35.081094 22228 init.cc:377] Logged in from keytab as oryx/127.14.41.66@KRBTEST.COM (short username oryx)
I20260925 10:08:35.081524 22228 webserver.cc:460] Webserver started at http://127.14.41.66:39467/ using document root <none> and password file <none>
I20260925 10:08:35.081774 22228 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:35.081823 22228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:35.082005 22228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:35.083118 22228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "50394402cef84a10b34012b8b108c29d"
format_stamp: "Formatted at 2026-09-25 10:08:35 on dist-test-slave-s6tm"
server_key: "2d034205e1136eeb8e20f5b8ee2b60a6"
server_key_iv: "d4f4533b1f523792f91304c2b0d1f4ce"
server_key_version: "encryptionkey@0"
I20260925 10:08:35.083498 22228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "50394402cef84a10b34012b8b108c29d"
format_stamp: "Formatted at 2026-09-25 10:08:35 on dist-test-slave-s6tm"
server_key: "2d034205e1136eeb8e20f5b8ee2b60a6"
server_key_iv: "d4f4533b1f523792f91304c2b0d1f4ce"
server_key_version: "encryptionkey@0"
I20260925 10:08:35.084961 22228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:08:35.085820 22243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:35.086047 22228 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:35.086141 22228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "50394402cef84a10b34012b8b108c29d"
format_stamp: "Formatted at 2026-09-25 10:08:35 on dist-test-slave-s6tm"
server_key: "2d034205e1136eeb8e20f5b8ee2b60a6"
server_key_iv: "d4f4533b1f523792f91304c2b0d1f4ce"
server_key_version: "encryptionkey@0"
I20260925 10:08:35.086198 22228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:35.095302 22228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:35.097414 22228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:35.097532 22228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:35.097997 22228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:35.098035 22228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:35.098059 22228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:35.098106 22228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:35.104990 22228 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:37351
I20260925 10:08:35.105069 22358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:37351 every 8 connection(s)
I20260925 10:08:35.105422 22228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:08:35 dist-test-slave-s6tm krb5kdc[21971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330915, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM
I20260925 10:08:35.109091 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.098603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:45335 (local address 127.14.41.126:34013)
0925 10:08:35.098664 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:35.098665 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:35.098948 (+ 283us) server_negotiation.cc:407] Connection header received
0925 10:08:35.099614 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:35.099617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:35.099632 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:35.099708 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:35.100430 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.101291 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.102919 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.103235 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.103338 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.106062 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:35.106063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:35.106064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:35.106071 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:35.107707 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:35.108137 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:35.108138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:35.108138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:35.108188 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:35.108481 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:35.108482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:35.108482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:35.108596 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:35.108647 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:35.108896 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 10:08:35.109027 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 10:08:35.109179 22252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.098594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:34013 (local address 127.14.41.66:45335)
0925 10:08:35.098801 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:35.098850 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:35.099485 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:35.099788 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:35.099793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:35.100138 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:35.100338 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.100343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.101408 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.101409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.102830 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.102833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.103349 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.103350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.103489 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.104175 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:35.104185 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:35.105967 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:35.107786 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:35.107791 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:35.107792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:35.108075 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:35.108246 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:35.108247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:35.108247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:35.108408 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:35.108653 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:35.108659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:35.108776 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 10:08:35.108855 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 10:08:35.108983 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":101,"threads_started":1}
I20260925 10:08:35.109676 22359 heartbeater.cc:347] Connected to a master server at 127.14.41.126:34013
I20260925 10:08:35.109757 22359 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:35.110028 22359 heartbeater.cc:511] Master 127.14.41.126:34013 requested a full tablet report, sending...
I20260925 10:08:35.110450 22016 ts_manager.cc:194] Registered new tserver with Master: 50394402cef84a10b34012b8b108c29d (127.14.41.66:37351)
I20260925 10:08:35.110991 22016 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:45335
I20260925 10:08:35.112850 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 22228
I20260925 10:08:35.112974 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:08:35.113194 14501 external_mini_cluster.cc:1428] Setting key 0729682fcb3944c1a40adf92c4014a8c
I20260925 10:08:35.117345 22365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.110715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:34013 (local address 127.14.41.66:53013)
0925 10:08:35.110846 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:35.110856 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:35.110916 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:35.111064 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:35.111065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:35.111293 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:35.111363 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.111365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.112197 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.112198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.112781 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.112783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.113358 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.113360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.113429 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.113978 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:35.113982 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:35.114279 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:35.116078 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:35.116079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:35.116080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:35.116331 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:35.116521 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:35.116522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:35.116522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:35.116596 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:35.117055 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:35.117056 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:35.117097 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 10:08:35.117174 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 10:08:35.117268 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260925 10:08:35.117487 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.110803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:53013 (local address 127.14.41.126:34013)
0925 10:08:35.110845 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:35.110846 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:35.110972 (+ 126us) server_negotiation.cc:407] Connection header received
0925 10:08:35.110989 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:35.110990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:35.110998 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:35.111035 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:35.111449 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.112114 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.112855 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.113225 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.113301 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.114365 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:35.114367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:35.114367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:35.114375 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:35.115995 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:35.116405 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:35.116406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:35.116406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:35.116454 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:35.116692 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:35.116693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:35.116693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:35.116980 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:35.117047 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:35.117255 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 10:08:35.117350 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for oryx/127.14.41.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.41.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.67.keytab.
Entry for principal oryx/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.67.keytab.
I20260925 10:08:35.143991 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:34013
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.67.keytab
--principal=oryx/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:35.229677 22368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:35.229940 22368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:35.229969 22368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:35.231184 22368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:35.231225 22368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:35.231251 22368 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:08:35.231271 22368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:08:35.232800 22368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.67.keytab
--principal=oryx/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.14.41.126:34013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:35.232993 22368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:35.233173 22368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:35.238451 22373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:35.238509 22368 server_base.cc:1034] running on GCE node
W20260925 10:08:35.238452 22376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:35.238451 22374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:35.239131 22368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:35.239369 22368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:35.240522 22368 hybrid_clock.cc:648] HybridClock initialized: now 1790330915240510 us; error 31 us; skew 500 ppm
Sep 25 10:08:35 dist-test-slave-s6tm krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330915, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:35.243445 22368 init.cc:377] Logged in from keytab as oryx/127.14.41.67@KRBTEST.COM (short username oryx)
I20260925 10:08:35.243991 22368 webserver.cc:460] Webserver started at http://127.14.41.67:36413/ using document root <none> and password file <none>
I20260925 10:08:35.244215 22368 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:35.244270 22368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:35.244371 22368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:35.245505 22368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "47532f24ee7641f994825b0a7d3f1682"
format_stamp: "Formatted at 2026-09-25 10:08:35 on dist-test-slave-s6tm"
server_key: "f2b6dc7d5bc62cb72d95832a64b1f7ec"
server_key_iv: "888564062e74c8c2ab69e357eabcac32"
server_key_version: "encryptionkey@0"
I20260925 10:08:35.245829 22368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "47532f24ee7641f994825b0a7d3f1682"
format_stamp: "Formatted at 2026-09-25 10:08:35 on dist-test-slave-s6tm"
server_key: "f2b6dc7d5bc62cb72d95832a64b1f7ec"
server_key_iv: "888564062e74c8c2ab69e357eabcac32"
server_key_version: "encryptionkey@0"
I20260925 10:08:35.247151 22368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:08:35.247989 22383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:35.248252 22368 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:35.248373 22368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "47532f24ee7641f994825b0a7d3f1682"
format_stamp: "Formatted at 2026-09-25 10:08:35 on dist-test-slave-s6tm"
server_key: "f2b6dc7d5bc62cb72d95832a64b1f7ec"
server_key_iv: "888564062e74c8c2ab69e357eabcac32"
server_key_version: "encryptionkey@0"
I20260925 10:08:35.248451 22368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:35.259536 22368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:35.261710 22368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:35.261896 22368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:35.262432 22368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:35.262476 22368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:35.262501 22368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:35.262542 22368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:35.269102 22368 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:46341
I20260925 10:08:35.269248 22498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:46341 every 8 connection(s)
I20260925 10:08:35.269452 22368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
Sep 25 10:08:35 dist-test-slave-s6tm krb5kdc[21971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330915, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM
I20260925 10:08:35.273655 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.262967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:39939 (local address 127.14.41.126:34013)
0925 10:08:35.263041 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:35.263044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:35.263256 (+ 212us) server_negotiation.cc:407] Connection header received
0925 10:08:35.263759 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:35.263761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:35.263776 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:35.263872 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:35.264893 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.265742 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.267410 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.267761 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.267863 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.270649 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:35.270651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:35.270651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:35.270659 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:35.272132 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:35.272617 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:35.272619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:35.272619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:35.272690 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:35.272969 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:35.272970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:35.272970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:35.273099 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:35.273176 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:35.273478 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 10:08:35.273561 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:08:35.273838 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.262959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:34013 (local address 127.14.41.67:39939)
0925 10:08:35.263124 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:35.263167 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:35.263662 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:35.264044 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:35.264049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:35.264467 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:35.264770 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.264779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.265869 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.265870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.267309 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.267312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.267875 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.267876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.268053 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.268691 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:35.268708 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:35.270529 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:35.272215 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:35.272223 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:35.272225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:35.272548 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:35.272775 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:35.272776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:35.272776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:35.272893 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:35.273167 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:35.273173 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:35.273317 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 10:08:35.273430 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 10:08:35.273603 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260925 10:08:35.274394 22499 heartbeater.cc:347] Connected to a master server at 127.14.41.126:34013
I20260925 10:08:35.274515 22499 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:35.274711 22499 heartbeater.cc:511] Master 127.14.41.126:34013 requested a full tablet report, sending...
I20260925 10:08:35.275143 22016 ts_manager.cc:194] Registered new tserver with Master: 47532f24ee7641f994825b0a7d3f1682 (127.14.41.67:46341)
I20260925 10:08:35.275586 22016 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:39939
I20260925 10:08:35.279273 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 22368
I20260925 10:08:35.279417 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:08:35.279628 14501 external_mini_cluster.cc:1428] Setting key d89cf65771ec069d07bfa9004e9bddc6
I20260925 10:08:35.280722 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:08:35.281544 22505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.275492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:34013 (local address 127.14.41.67:59395)
0925 10:08:35.275634 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:35.275647 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:35.275711 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:35.275857 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:35.275858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:35.276126 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:35.276195 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.276197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.276858 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.276859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.277350 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.277352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.277666 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.277667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.277718 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.278152 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:35.278156 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:35.278439 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:35.280312 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:35.280313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:35.280313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:35.280513 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:35.280882 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:35.280883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:35.280883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:35.280957 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:35.281239 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:35.281239 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:35.281280 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 10:08:35.281377 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 10:08:35.281467 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260925 10:08:35.281625 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.275552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:59395 (local address 127.14.41.126:34013)
0925 10:08:35.275598 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:35.275600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:35.275761 (+ 161us) server_negotiation.cc:407] Connection header received
0925 10:08:35.275784 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:35.275785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:35.275795 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:35.275853 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:35.276315 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.276768 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.277421 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.277600 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.277652 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.278523 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:35.278525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:35.278525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:35.278531 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:35.280211 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:35.280599 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:35.280600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:35.280600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:35.280662 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:35.281031 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:35.281032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:35.281033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:35.281171 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:35.281246 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:35.281405 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 10:08:35.281493 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 25 10:08:35 dist-test-slave-s6tm krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330915, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:08:35 dist-test-slave-s6tm krb5kdc[21971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:35 dist-test-slave-s6tm krb5kdc[21971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:08:35.304544 22084 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:35.298348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.14.41.126:34013)
0925 10:08:35.298415 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:35.298417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:35.298426 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:08:35.298526 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:35.298527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:35.298536 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:35.298596 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:35.299247 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.299790 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.300668 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.300869 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.300971 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.304439 (+ 3468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43846: BlockingRecv error: recv got EOF from 127.0.0.1:43846 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 10:08:35 dist-test-slave-s6tm krb5kdc[21971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330915, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM
I20260925 10:08:35.312820 22084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.305401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43860 (local address 127.14.41.126:34013)
0925 10:08:35.305497 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:35.305499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:35.305585 (+ 86us) server_negotiation.cc:407] Connection header received
0925 10:08:35.305687 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:35.305687 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:35.305697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:35.305750 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:35.306287 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.306773 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.307498 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.307696 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.307769 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.310079 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:35.310080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:35.310080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:35.310087 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:35.311500 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:35.312035 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:35.312036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:35.312036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:35.312089 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:35.312340 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:35.312341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:35.312341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:35.312455 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:35.312538 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:35.312657 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 10:08:35.312750 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:08:35.314713 22016 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:08:35.316637 22016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:08:35.323565 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.319400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51572 (local address 127.14.41.65:42479)
0925 10:08:35.319570 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:35.319576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:35.319586 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:08:35.319625 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:35.319628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:35.319642 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:35.319892 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:35.320092 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.320923 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.322356 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.323118 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.323191 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.323325 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:35.323410 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 10:08:35.323460 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260925 10:08:35.323563 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.319127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:42479 (local address 127.0.0.1:51572)
0925 10:08:35.319297 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:35.319338 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:35.319404 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:35.319898 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:35.319901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:35.319904 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:35.320002 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.320004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.321029 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.321030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.322268 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.322271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.323193 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.323194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.323298 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.323317 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 10:08:35.323396 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 10:08:35.323459 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":1,"thread_start_us":60,"threads_started":1}
I20260925 10:08:35.323740 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.319140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:46341 (local address 127.0.0.1:53644)
0925 10:08:35.319297 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:35.319338 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:35.319404 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:35.320157 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:35.320157 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:35.320159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:35.320193 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.320195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.321548 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.321549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.322579 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.322582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.323501 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.323502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.323566 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.323571 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:08:35.323601 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 10:08:35.323646 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260925 10:08:35.324168 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.319577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53644 (local address 127.14.41.67:46341)
0925 10:08:35.319863 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:35.319866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:35.319876 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:08:35.320037 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:35.320039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:35.320059 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:35.320263 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:35.320328 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.321429 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.322674 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.323407 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.323914 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.324028 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:35.324054 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 10:08:35.324091 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
I20260925 10:08:35.324318 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.319140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:37351 (local address 127.0.0.1:40034)
0925 10:08:35.319509 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:35.319518 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:35.319556 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:35.320089 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:35.320090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:35.320093 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:35.320156 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.320158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.321481 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.321482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.322526 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.322528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.323450 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.323451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.323501 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.323504 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:35.323550 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 10:08:35.323589 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":44,"thread_start_us":125,"threads_started":1}
I20260925 10:08:35.324401 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.319405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40034 (local address 127.14.41.66:37351)
0925 10:08:35.319799 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:35.319804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:35.319816 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:08:35.319857 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:35.319860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:35.319877 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:35.320008 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:35.320251 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.321387 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.322621 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.323277 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.323343 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.323418 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:35.323629 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 10:08:35.324108 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":67,"threads_started":1}
I20260925 10:08:35.324519 22153 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2d4d69c87a49cfbf1894e1933e46ea (DEFAULT_TABLE table=test-table [id=932ac25f007c4c89b33d1e14066f9cb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:35.324926 22153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2d4d69c87a49cfbf1894e1933e46ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:35.324880 22433 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2d4d69c87a49cfbf1894e1933e46ea (DEFAULT_TABLE table=test-table [id=932ac25f007c4c89b33d1e14066f9cb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:35.324970 22293 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2d4d69c87a49cfbf1894e1933e46ea (DEFAULT_TABLE table=test-table [id=932ac25f007c4c89b33d1e14066f9cb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:35.325227 22433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2d4d69c87a49cfbf1894e1933e46ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:35.325275 22293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2d4d69c87a49cfbf1894e1933e46ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:35.328092 22527 tablet_bootstrap.cc:492] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f: Bootstrap starting.
I20260925 10:08:35.328794 22527 tablet_bootstrap.cc:654] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:35.329062 22528 tablet_bootstrap.cc:492] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682: Bootstrap starting.
I20260925 10:08:35.329301 22527 log.cc:824] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:35.329679 22529 tablet_bootstrap.cc:492] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d: Bootstrap starting.
I20260925 10:08:35.329908 22528 tablet_bootstrap.cc:654] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:35.330104 22527 tablet_bootstrap.cc:492] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f: No bootstrap required, opened a new log
I20260925 10:08:35.330200 22527 ts_tablet_manager.cc:1397] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 10:08:35.330250 22528 log.cc:824] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:35.330322 22529 tablet_bootstrap.cc:654] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:35.330619 22529 log.cc:824] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:35.331017 22528 tablet_bootstrap.cc:492] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682: No bootstrap required, opened a new log
I20260925 10:08:35.331101 22528 ts_tablet_manager.cc:1397] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:08:35.331459 22529 tablet_bootstrap.cc:492] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d: No bootstrap required, opened a new log
I20260925 10:08:35.331537 22529 ts_tablet_manager.cc:1397] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:08:35.332171 22527 raft_consensus.cc:348] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } } peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } }
I20260925 10:08:35.332363 22527 raft_consensus.cc:374] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:35.332409 22527 raft_consensus.cc:729] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d20d5f21224c2091e691e77fce586f, State: Initialized, Role: FOLLOWER
I20260925 10:08:35.332523 22527 consensus_queue.cc:260] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } } peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } }
I20260925 10:08:35.332815 22527 ts_tablet_manager.cc:1428] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 10:08:35.332949 22219 heartbeater.cc:503] Master 127.14.41.126:34013 was elected leader, sending a full tablet report...
I20260925 10:08:35.333154 22528 raft_consensus.cc:348] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } } peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } }
I20260925 10:08:35.333324 22528 raft_consensus.cc:374] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:35.333359 22528 raft_consensus.cc:729] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47532f24ee7641f994825b0a7d3f1682, State: Initialized, Role: FOLLOWER
I20260925 10:08:35.333470 22528 consensus_queue.cc:260] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } } peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } }
I20260925 10:08:35.333719 22528 ts_tablet_manager.cc:1428] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 10:08:35.333586 22529 raft_consensus.cc:348] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } } peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } }
I20260925 10:08:35.333767 22529 raft_consensus.cc:374] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:35.333817 22529 raft_consensus.cc:729] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50394402cef84a10b34012b8b108c29d, State: Initialized, Role: FOLLOWER
I20260925 10:08:35.333838 22499 heartbeater.cc:503] Master 127.14.41.126:34013 was elected leader, sending a full tablet report...
I20260925 10:08:35.333931 22529 consensus_queue.cc:260] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } } peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } }
I20260925 10:08:35.334190 22529 ts_tablet_manager.cc:1428] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:08:35.334201 22359 heartbeater.cc:503] Master 127.14.41.126:34013 was elected leader, sending a full tablet report...
W20260925 10:08:35.356542 22360 tablet.cc:2369] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:08:35.434355 22220 tablet.cc:2369] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:08:35.520509 22500 tablet.cc:2369] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:08:35.526432 22534 raft_consensus.cc:482] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:35.526562 22534 raft_consensus.cc:504] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } } peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } }
I20260925 10:08:35.526954 22534 leader_election.cc:288] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79d20d5f21224c2091e691e77fce586f (127.14.41.65:42479), 50394402cef84a10b34012b8b108c29d (127.14.41.66:37351)
I20260925 10:08:35.530174 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.527275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:56403 (local address 127.14.41.66:37351)
0925 10:08:35.527351 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:35.527353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:35.527389 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:08:35.527485 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:35.527487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:35.527498 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:35.527570 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:35.527825 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.528410 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.529416 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.529796 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.529851 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.529858 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:35.530054 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 10:08:35.530100 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:08:35.530184 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.527192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:37351 (local address 127.14.41.67:56403)
0925 10:08:35.527326 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:35.527338 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:35.527407 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:35.527601 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:35.527602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:35.527604 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:35.527684 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.527687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.528493 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.528494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.529321 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.529324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.529879 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.529880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.529980 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.529983 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:35.530036 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 10:08:35.530103 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":18}
I20260925 10:08:35.530210 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.527267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:55137 (local address 127.14.41.65:42479)
0925 10:08:35.527353 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:35.527356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:35.527601 (+ 245us) server_negotiation.cc:407] Connection header received
0925 10:08:35.527634 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:35.527635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:35.527645 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:35.527812 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:35.528006 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.528496 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.529469 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.529860 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.529933 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.529941 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:35.530106 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 10:08:35.530145 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:08:35.530225 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.527195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:42479 (local address 127.14.41.67:55137)
0925 10:08:35.527510 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:35.527520 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:35.527578 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:35.527818 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:35.527819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:35.527821 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:35.527898 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.527899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.528574 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.528575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.529341 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.529344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.529944 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.529945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.530004 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.530007 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:35.530084 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 10:08:35.530121 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":113,"thread_start_us":120,"threads_started":1}
I20260925 10:08:35.530589 22313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2d4d69c87a49cfbf1894e1933e46ea" candidate_uuid: "47532f24ee7641f994825b0a7d3f1682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50394402cef84a10b34012b8b108c29d" is_pre_election: true
I20260925 10:08:35.530769 22313 raft_consensus.cc:2454] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47532f24ee7641f994825b0a7d3f1682 in term 0.
I20260925 10:08:35.530922 22173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2d4d69c87a49cfbf1894e1933e46ea" candidate_uuid: "47532f24ee7641f994825b0a7d3f1682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79d20d5f21224c2091e691e77fce586f" is_pre_election: true
I20260925 10:08:35.531082 22173 raft_consensus.cc:2454] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47532f24ee7641f994825b0a7d3f1682 in term 0.
I20260925 10:08:35.531062 22384 leader_election.cc:302] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47532f24ee7641f994825b0a7d3f1682, 50394402cef84a10b34012b8b108c29d; no voters:
I20260925 10:08:35.531203 22534 raft_consensus.cc:2781] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:35.531286 22534 raft_consensus.cc:482] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:35.531325 22534 raft_consensus.cc:3037] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:35.532413 22534 raft_consensus.cc:504] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } } peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } }
I20260925 10:08:35.532583 22534 leader_election.cc:288] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [CANDIDATE]: Term 1 election: Requested vote from peers 79d20d5f21224c2091e691e77fce586f (127.14.41.65:42479), 50394402cef84a10b34012b8b108c29d (127.14.41.66:37351)
I20260925 10:08:35.532778 22313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2d4d69c87a49cfbf1894e1933e46ea" candidate_uuid: "47532f24ee7641f994825b0a7d3f1682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50394402cef84a10b34012b8b108c29d"
I20260925 10:08:35.532778 22173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2d4d69c87a49cfbf1894e1933e46ea" candidate_uuid: "47532f24ee7641f994825b0a7d3f1682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79d20d5f21224c2091e691e77fce586f"
I20260925 10:08:35.532866 22313 raft_consensus.cc:3037] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:35.532866 22173 raft_consensus.cc:3037] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:35.533780 22313 raft_consensus.cc:2454] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47532f24ee7641f994825b0a7d3f1682 in term 1.
I20260925 10:08:35.533780 22173 raft_consensus.cc:2454] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47532f24ee7641f994825b0a7d3f1682 in term 1.
I20260925 10:08:35.533967 22384 leader_election.cc:302] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47532f24ee7641f994825b0a7d3f1682, 50394402cef84a10b34012b8b108c29d; no voters:
I20260925 10:08:35.534071 22534 raft_consensus.cc:2781] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:35.534229 22534 raft_consensus.cc:686] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [term 1 LEADER]: Becoming Leader. State: Replica: 47532f24ee7641f994825b0a7d3f1682, State: Running, Role: LEADER
I20260925 10:08:35.534345 22534 consensus_queue.cc:237] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } } peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } }
I20260925 10:08:35.535277 22015 catalog_manager.cc:5495] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47532f24ee7641f994825b0a7d3f1682 (127.14.41.67). New cstate: current_term: 1 leader_uuid: "47532f24ee7641f994825b0a7d3f1682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } health_report { overall_health: UNKNOWN } } }
I20260925 10:08:35.556175 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.553482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53656 (local address 127.14.41.67:46341)
0925 10:08:35.553551 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:35.553553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:35.553561 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:08:35.553599 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:35.553600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:35.553618 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:35.553680 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:35.553860 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.554347 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.555171 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.555383 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.555449 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.555654 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:35.555812 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:35.555911 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:35.556020 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 10:08:35.556062 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:08:35.558284 22173 raft_consensus.cc:1261] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f [term 1 FOLLOWER]: Refusing update from remote peer 47532f24ee7641f994825b0a7d3f1682: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:35.558284 22313 raft_consensus.cc:1261] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d [term 1 FOLLOWER]: Refusing update from remote peer 47532f24ee7641f994825b0a7d3f1682: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:35.558697 22537 consensus_queue.cc:1035] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:35.558833 22534 consensus_queue.cc:1035] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:35.561983 22541 mvcc.cc:205] Tried to move back new op lower bound from 7333195430123593728 to 7333195430029430784. Current Snapshot: MvccSnapshot[applied={T|T < 7333195430123593728 or (T in {7333195430123593728})}]
I20260925 10:08:35.562031 22542 mvcc.cc:205] Tried to move back new op lower bound from 7333195430123593728 to 7333195430029430784. Current Snapshot: MvccSnapshot[applied={T|T < 7333195430123593728 or (T in {7333195430123593728})}]
I20260925 10:08:35.564250 22015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:35.564355 22015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:35.565402 22015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de: Sending DeleteTablet for 3 replicas of tablet 7a2d4d69c87a49cfbf1894e1933e46ea
I20260925 10:08:35.565786 22433 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a2d4d69c87a49cfbf1894e1933e46ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:35 UTC) from {username='oryx', principal='oryx/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:53644
I20260925 10:08:35.565883 22293 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a2d4d69c87a49cfbf1894e1933e46ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:35 UTC) from {username='oryx', principal='oryx/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:40034
I20260925 10:08:35.566093 22153 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a2d4d69c87a49cfbf1894e1933e46ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:35 UTC) from {username='oryx', principal='oryx/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:51572
I20260925 10:08:35.566238 22550 tablet_replica.cc:330] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682: stopping tablet replica
I20260925 10:08:35.566352 22550 raft_consensus.cc:2229] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:08:35.566356 22552 tablet_replica.cc:330] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f: stopping tablet replica
I20260925 10:08:35.566494 22550 raft_consensus.cc:2258] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:35.566425 22015 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:08:35.566573 22015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:08:35.566821 22552 raft_consensus.cc:2229] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:35.566920 22551 tablet_replica.cc:330] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d: stopping tablet replica
I20260925 10:08:35.567029 22551 raft_consensus.cc:2229] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:35.567106 22552 raft_consensus.cc:2258] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:35.567138 22551 raft_consensus.cc:2258] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:35.567670 22550 ts_tablet_manager.cc:1905] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:35.567669 22552 ts_tablet_manager.cc:1905] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:35.568252 22433 tablet_service.cc:1467] Processing CreateTablet for tablet 60f0d71a585d4fdc907c112445b3adb7 (DEFAULT_TABLE table=test-table [id=3d8b32b7fb0e44cb9000aee26642c17b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:35.568385 22433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f0d71a585d4fdc907c112445b3adb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:35.569644 22552 ts_tablet_manager.cc:1918] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:08:35.569644 22550 ts_tablet_manager.cc:1918] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:08:35.569726 22552 log.cc:1196] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-0/wal/wals/7a2d4d69c87a49cfbf1894e1933e46ea
I20260925 10:08:35.569708 22528 tablet_bootstrap.cc:492] T 60f0d71a585d4fdc907c112445b3adb7 P 47532f24ee7641f994825b0a7d3f1682: Bootstrap starting.
I20260925 10:08:35.569860 22550 log.cc:1196] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/7a2d4d69c87a49cfbf1894e1933e46ea
I20260925 10:08:35.570025 22552 ts_tablet_manager.cc:1939] T 7a2d4d69c87a49cfbf1894e1933e46ea P 79d20d5f21224c2091e691e77fce586f: Deleting consensus metadata
I20260925 10:08:35.570307 22528 tablet_bootstrap.cc:654] T 60f0d71a585d4fdc907c112445b3adb7 P 47532f24ee7641f994825b0a7d3f1682: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:35.570325 22550 ts_tablet_manager.cc:1939] T 7a2d4d69c87a49cfbf1894e1933e46ea P 47532f24ee7641f994825b0a7d3f1682: Deleting consensus metadata
I20260925 10:08:35.570926 22293 tablet_service.cc:1467] Processing CreateTablet for tablet 60f0d71a585d4fdc907c112445b3adb7 (DEFAULT_TABLE table=test-table [id=3d8b32b7fb0e44cb9000aee26642c17b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:35.570935 22001 catalog_manager.cc:4841] TS 47532f24ee7641f994825b0a7d3f1682 (127.14.41.67:46341): tablet 7a2d4d69c87a49cfbf1894e1933e46ea (table test-table [id=932ac25f007c4c89b33d1e14066f9cb3]) successfully deleted
I20260925 10:08:35.571063 22293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f0d71a585d4fdc907c112445b3adb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:35.571092 22002 catalog_manager.cc:4841] TS 79d20d5f21224c2091e691e77fce586f (127.14.41.65:42479): tablet 7a2d4d69c87a49cfbf1894e1933e46ea (table test-table [id=932ac25f007c4c89b33d1e14066f9cb3]) successfully deleted
I20260925 10:08:35.571278 22153 tablet_service.cc:1467] Processing CreateTablet for tablet 60f0d71a585d4fdc907c112445b3adb7 (DEFAULT_TABLE table=test-table [id=3d8b32b7fb0e44cb9000aee26642c17b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:35.571389 22153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f0d71a585d4fdc907c112445b3adb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:35.572149 22528 tablet_bootstrap.cc:492] T 60f0d71a585d4fdc907c112445b3adb7 P 47532f24ee7641f994825b0a7d3f1682: No bootstrap required, opened a new log
I20260925 10:08:35.572217 22528 ts_tablet_manager.cc:1397] T 60f0d71a585d4fdc907c112445b3adb7 P 47532f24ee7641f994825b0a7d3f1682: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 10:08:35.572292 22529 tablet_bootstrap.cc:492] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d: Bootstrap starting.
I20260925 10:08:35.572430 22551 ts_tablet_manager.cc:1905] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:35.572400 22528 raft_consensus.cc:348] T 60f0d71a585d4fdc907c112445b3adb7 P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:35.572489 22528 raft_consensus.cc:374] T 60f0d71a585d4fdc907c112445b3adb7 P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:35.572523 22528 raft_consensus.cc:729] T 60f0d71a585d4fdc907c112445b3adb7 P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47532f24ee7641f994825b0a7d3f1682, State: Initialized, Role: FOLLOWER
I20260925 10:08:35.572566 22528 consensus_queue.cc:260] T 60f0d71a585d4fdc907c112445b3adb7 P 47532f24ee7641f994825b0a7d3f1682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:35.572661 22528 ts_tablet_manager.cc:1428] T 60f0d71a585d4fdc907c112445b3adb7 P 47532f24ee7641f994825b0a7d3f1682: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 10:08:35.572779 22529 tablet_bootstrap.cc:654] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:35.574076 22527 tablet_bootstrap.cc:492] T 60f0d71a585d4fdc907c112445b3adb7 P 79d20d5f21224c2091e691e77fce586f: Bootstrap starting.
I20260925 10:08:35.574384 22529 tablet_bootstrap.cc:492] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d: No bootstrap required, opened a new log
I20260925 10:08:35.574448 22529 ts_tablet_manager.cc:1397] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:08:35.574507 22527 tablet_bootstrap.cc:654] T 60f0d71a585d4fdc907c112445b3adb7 P 79d20d5f21224c2091e691e77fce586f: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:35.574615 22529 raft_consensus.cc:348] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:35.574699 22529 raft_consensus.cc:374] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:35.574719 22529 raft_consensus.cc:729] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50394402cef84a10b34012b8b108c29d, State: Initialized, Role: FOLLOWER
I20260925 10:08:35.574766 22529 consensus_queue.cc:260] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:35.574852 22551 ts_tablet_manager.cc:1918] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:08:35.574887 22529 ts_tablet_manager.cc:1428] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:35.574928 22551 log.cc:1196] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/7a2d4d69c87a49cfbf1894e1933e46ea
I20260925 10:08:35.575215 22551 ts_tablet_manager.cc:1939] T 7a2d4d69c87a49cfbf1894e1933e46ea P 50394402cef84a10b34012b8b108c29d: Deleting consensus metadata
I20260925 10:08:35.575656 22000 catalog_manager.cc:4841] TS 50394402cef84a10b34012b8b108c29d (127.14.41.66:37351): tablet 7a2d4d69c87a49cfbf1894e1933e46ea (table test-table [id=932ac25f007c4c89b33d1e14066f9cb3]) successfully deleted
I20260925 10:08:35.575958 22527 tablet_bootstrap.cc:492] T 60f0d71a585d4fdc907c112445b3adb7 P 79d20d5f21224c2091e691e77fce586f: No bootstrap required, opened a new log
I20260925 10:08:35.576045 22527 ts_tablet_manager.cc:1397] T 60f0d71a585d4fdc907c112445b3adb7 P 79d20d5f21224c2091e691e77fce586f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:08:35.576205 22527 raft_consensus.cc:348] T 60f0d71a585d4fdc907c112445b3adb7 P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:35.576284 22527 raft_consensus.cc:374] T 60f0d71a585d4fdc907c112445b3adb7 P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:35.576318 22527 raft_consensus.cc:729] T 60f0d71a585d4fdc907c112445b3adb7 P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d20d5f21224c2091e691e77fce586f, State: Initialized, Role: FOLLOWER
I20260925 10:08:35.576367 22527 consensus_queue.cc:260] T 60f0d71a585d4fdc907c112445b3adb7 P 79d20d5f21224c2091e691e77fce586f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:35.576673 22527 ts_tablet_manager.cc:1428] T 60f0d71a585d4fdc907c112445b3adb7 P 79d20d5f21224c2091e691e77fce586f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 10:08:35.939810 22535 raft_consensus.cc:482] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:35.939987 22535 raft_consensus.cc:504] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:35.940300 22535 leader_election.cc:288] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79d20d5f21224c2091e691e77fce586f (127.14.41.65:42479), 47532f24ee7641f994825b0a7d3f1682 (127.14.41.67:46341)
I20260925 10:08:35.943446 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.940674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:41705 (local address 127.14.41.67:46341)
0925 10:08:35.940732 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:35.940734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:35.940779 (+ 45us) server_negotiation.cc:407] Connection header received
0925 10:08:35.940864 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:35.940865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:35.940877 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:35.940968 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:35.941141 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.941652 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.942651 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.943070 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.943154 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.943163 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:35.943332 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 10:08:35.943373 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 10:08:35.943524 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.940608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:56139 (local address 127.14.41.65:42479)
0925 10:08:35.940661 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:35.940664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:35.940745 (+ 81us) server_negotiation.cc:407] Connection header received
0925 10:08:35.940824 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:35.940825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:35.940837 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:35.941011 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:35.941182 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.941699 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.942716 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.943124 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.943214 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.943223 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:35.943399 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 10:08:35.943440 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:08:35.943576 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.940524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:42479 (local address 127.14.41.66:56139)
0925 10:08:35.940673 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:35.940685 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:35.940759 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:35.941021 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:35.941022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:35.941024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:35.941095 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.941096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.941791 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.941792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.942603 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.942605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.943213 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.943214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.943315 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.943319 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:08:35.943377 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 10:08:35.943435 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
I20260925 10:08:35.943468 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:35.940524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:46341 (local address 127.14.41.66:41705)
0925 10:08:35.940703 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:35.940717 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:35.940800 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:35.940976 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:35.940977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:35.940980 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:35.941058 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.941061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.941746 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.941747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.942542 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:35.942544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:35.943153 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:35.943154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:35.943232 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:35.943236 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:08:35.943318 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 10:08:35.943364 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":2,"thread_start_us":73,"threads_started":1}
I20260925 10:08:35.943801 22173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f0d71a585d4fdc907c112445b3adb7" candidate_uuid: "50394402cef84a10b34012b8b108c29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79d20d5f21224c2091e691e77fce586f" is_pre_election: true
I20260925 10:08:35.943949 22173 raft_consensus.cc:2454] T 60f0d71a585d4fdc907c112445b3adb7 P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50394402cef84a10b34012b8b108c29d in term 0.
I20260925 10:08:35.944017 22453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f0d71a585d4fdc907c112445b3adb7" candidate_uuid: "50394402cef84a10b34012b8b108c29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47532f24ee7641f994825b0a7d3f1682" is_pre_election: true
I20260925 10:08:35.944154 22453 raft_consensus.cc:2454] T 60f0d71a585d4fdc907c112445b3adb7 P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50394402cef84a10b34012b8b108c29d in term 0.
I20260925 10:08:35.944378 22246 leader_election.cc:302] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50394402cef84a10b34012b8b108c29d, 79d20d5f21224c2091e691e77fce586f; no voters:
I20260925 10:08:35.944521 22535 raft_consensus.cc:2781] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:35.944562 22535 raft_consensus.cc:482] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:35.944583 22535 raft_consensus.cc:3037] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:35.945356 22535 raft_consensus.cc:504] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:35.945510 22535 leader_election.cc:288] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [CANDIDATE]: Term 1 election: Requested vote from peers 79d20d5f21224c2091e691e77fce586f (127.14.41.65:42479), 47532f24ee7641f994825b0a7d3f1682 (127.14.41.67:46341)
I20260925 10:08:35.945686 22173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f0d71a585d4fdc907c112445b3adb7" candidate_uuid: "50394402cef84a10b34012b8b108c29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79d20d5f21224c2091e691e77fce586f"
I20260925 10:08:35.945696 22453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f0d71a585d4fdc907c112445b3adb7" candidate_uuid: "50394402cef84a10b34012b8b108c29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47532f24ee7641f994825b0a7d3f1682"
I20260925 10:08:35.945771 22453 raft_consensus.cc:3037] T 60f0d71a585d4fdc907c112445b3adb7 P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:35.945771 22173 raft_consensus.cc:3037] T 60f0d71a585d4fdc907c112445b3adb7 P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:35.946542 22173 raft_consensus.cc:2454] T 60f0d71a585d4fdc907c112445b3adb7 P 79d20d5f21224c2091e691e77fce586f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50394402cef84a10b34012b8b108c29d in term 1.
I20260925 10:08:35.946709 22453 raft_consensus.cc:2454] T 60f0d71a585d4fdc907c112445b3adb7 P 47532f24ee7641f994825b0a7d3f1682 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50394402cef84a10b34012b8b108c29d in term 1.
I20260925 10:08:35.946740 22246 leader_election.cc:302] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50394402cef84a10b34012b8b108c29d, 79d20d5f21224c2091e691e77fce586f; no voters:
I20260925 10:08:35.946839 22535 raft_consensus.cc:2781] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:35.946970 22535 raft_consensus.cc:686] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [term 1 LEADER]: Becoming Leader. State: Replica: 50394402cef84a10b34012b8b108c29d, State: Running, Role: LEADER
I20260925 10:08:35.947042 22535 consensus_queue.cc:237] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:35.947808 22015 catalog_manager.cc:5495] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d reported cstate change: term changed from 0 to 1, leader changed from <none> to 50394402cef84a10b34012b8b108c29d (127.14.41.66). New cstate: current_term: 1 leader_uuid: "50394402cef84a10b34012b8b108c29d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } health_report { overall_health: UNKNOWN } } }
Sep 25 10:08:36 dist-test-slave-s6tm krb5kdc[21971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330914, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.126@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM
I20260925 10:08:36.057075 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:36.049031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43864 (local address 127.14.41.126:34013)
0925 10:08:36.049212 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:36.049214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:36.049225 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:08:36.049252 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:36.049253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:36.049263 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:36.049372 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:36.049953 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.050468 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.051313 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.051530 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.051601 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:36.054113 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:36.054114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:36.054114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:36.054121 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:36.055593 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:36.056031 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:36.056032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:36.056033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:36.056092 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:36.056422 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:36.056424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:36.056424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:36.056544 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:36.056624 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:36.056774 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 10:08:36.056930 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
I20260925 10:08:36.057127 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:36.048860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:34013 (local address 127.0.0.1:43864)
0925 10:08:36.049066 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:36.049081 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:36.049161 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:36.049395 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:36.049396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:36.049708 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:36.049850 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:36.049858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.050561 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.050562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:36.051234 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:36.051236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.051613 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.051614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:36.051659 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:36.052326 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:36.052331 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:36.054011 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:36.055668 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:36.055675 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:36.055677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:36.055891 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:36.056182 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:36.056183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:36.056183 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:36.056244 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:36.056615 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:36.056623 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:36.056673 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 10:08:36.056750 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 10:08:36.056989 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":137,"thread_start_us":103,"threads_started":1}
I20260925 10:08:36.059357 22015 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:43864:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 10:08:36.059675 22015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:08:36.061398 22153 tablet_service.cc:1467] Processing CreateTablet for tablet 224e3ee58e6b43a0a98abfc3d1861e6f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b7a233b73f04469e9e78dee714d6ded1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:08:36.061407 22433 tablet_service.cc:1467] Processing CreateTablet for tablet 224e3ee58e6b43a0a98abfc3d1861e6f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b7a233b73f04469e9e78dee714d6ded1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:08:36.061407 22293 tablet_service.cc:1467] Processing CreateTablet for tablet 224e3ee58e6b43a0a98abfc3d1861e6f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b7a233b73f04469e9e78dee714d6ded1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:08:36.061569 22153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224e3ee58e6b43a0a98abfc3d1861e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:36.061574 22433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224e3ee58e6b43a0a98abfc3d1861e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:36.061594 22293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 224e3ee58e6b43a0a98abfc3d1861e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:36.063086 22529 tablet_bootstrap.cc:492] T 224e3ee58e6b43a0a98abfc3d1861e6f P 50394402cef84a10b34012b8b108c29d: Bootstrap starting.
I20260925 10:08:36.063308 22528 tablet_bootstrap.cc:492] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682: Bootstrap starting.
I20260925 10:08:36.063633 22529 tablet_bootstrap.cc:654] T 224e3ee58e6b43a0a98abfc3d1861e6f P 50394402cef84a10b34012b8b108c29d: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:36.063699 22528 tablet_bootstrap.cc:654] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:36.064020 22527 tablet_bootstrap.cc:492] T 224e3ee58e6b43a0a98abfc3d1861e6f P 79d20d5f21224c2091e691e77fce586f: Bootstrap starting.
I20260925 10:08:36.064285 22528 tablet_bootstrap.cc:492] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682: No bootstrap required, opened a new log
I20260925 10:08:36.064348 22528 ts_tablet_manager.cc:1397] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:36.064459 22527 tablet_bootstrap.cc:654] T 224e3ee58e6b43a0a98abfc3d1861e6f P 79d20d5f21224c2091e691e77fce586f: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:36.064496 22528 raft_consensus.cc:348] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.064602 22528 raft_consensus.cc:374] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:36.064628 22528 raft_consensus.cc:729] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47532f24ee7641f994825b0a7d3f1682, State: Initialized, Role: FOLLOWER
I20260925 10:08:36.064695 22528 consensus_queue.cc:260] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.064814 22528 ts_tablet_manager.cc:1428] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:36.064845 22545 tablet_replica.cc:439] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } } }
I20260925 10:08:36.064939 22545 tablet_replica.cc:442] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:08:36.065074 22529 tablet_bootstrap.cc:492] T 224e3ee58e6b43a0a98abfc3d1861e6f P 50394402cef84a10b34012b8b108c29d: No bootstrap required, opened a new log
I20260925 10:08:36.065138 22529 ts_tablet_manager.cc:1397] T 224e3ee58e6b43a0a98abfc3d1861e6f P 50394402cef84a10b34012b8b108c29d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 10:08:36.065318 22529 raft_consensus.cc:348] T 224e3ee58e6b43a0a98abfc3d1861e6f P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.065410 22529 raft_consensus.cc:374] T 224e3ee58e6b43a0a98abfc3d1861e6f P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:36.065440 22529 raft_consensus.cc:729] T 224e3ee58e6b43a0a98abfc3d1861e6f P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50394402cef84a10b34012b8b108c29d, State: Initialized, Role: FOLLOWER
I20260925 10:08:36.065486 22529 consensus_queue.cc:260] T 224e3ee58e6b43a0a98abfc3d1861e6f P 50394402cef84a10b34012b8b108c29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.065611 22529 ts_tablet_manager.cc:1428] T 224e3ee58e6b43a0a98abfc3d1861e6f P 50394402cef84a10b34012b8b108c29d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:36.065668 22535 tablet_replica.cc:439] T 224e3ee58e6b43a0a98abfc3d1861e6f P 50394402cef84a10b34012b8b108c29d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } } }
I20260925 10:08:36.065769 22535 tablet_replica.cc:442] T 224e3ee58e6b43a0a98abfc3d1861e6f P 50394402cef84a10b34012b8b108c29d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:08:36.066051 22527 tablet_bootstrap.cc:492] T 224e3ee58e6b43a0a98abfc3d1861e6f P 79d20d5f21224c2091e691e77fce586f: No bootstrap required, opened a new log
I20260925 10:08:36.066113 22527 ts_tablet_manager.cc:1397] T 224e3ee58e6b43a0a98abfc3d1861e6f P 79d20d5f21224c2091e691e77fce586f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:08:36.066298 22527 raft_consensus.cc:348] T 224e3ee58e6b43a0a98abfc3d1861e6f P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.066385 22527 raft_consensus.cc:374] T 224e3ee58e6b43a0a98abfc3d1861e6f P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:36.066414 22527 raft_consensus.cc:729] T 224e3ee58e6b43a0a98abfc3d1861e6f P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d20d5f21224c2091e691e77fce586f, State: Initialized, Role: FOLLOWER
I20260925 10:08:36.066455 22527 consensus_queue.cc:260] T 224e3ee58e6b43a0a98abfc3d1861e6f P 79d20d5f21224c2091e691e77fce586f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.066560 22527 ts_tablet_manager.cc:1428] T 224e3ee58e6b43a0a98abfc3d1861e6f P 79d20d5f21224c2091e691e77fce586f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:36.066635 22533 tablet_replica.cc:439] T 224e3ee58e6b43a0a98abfc3d1861e6f P 79d20d5f21224c2091e691e77fce586f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } } }
I20260925 10:08:36.066720 22533 tablet_replica.cc:442] T 224e3ee58e6b43a0a98abfc3d1861e6f P 79d20d5f21224c2091e691e77fce586f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:08:36.309661 22545 raft_consensus.cc:482] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:36.309813 22545 raft_consensus.cc:504] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.310058 22545 leader_election.cc:288] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79d20d5f21224c2091e691e77fce586f (127.14.41.65:42479), 50394402cef84a10b34012b8b108c29d (127.14.41.66:37351)
I20260925 10:08:36.310300 22313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224e3ee58e6b43a0a98abfc3d1861e6f" candidate_uuid: "47532f24ee7641f994825b0a7d3f1682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50394402cef84a10b34012b8b108c29d" is_pre_election: true
I20260925 10:08:36.310464 22313 raft_consensus.cc:2454] T 224e3ee58e6b43a0a98abfc3d1861e6f P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47532f24ee7641f994825b0a7d3f1682 in term 0.
I20260925 10:08:36.310495 22173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224e3ee58e6b43a0a98abfc3d1861e6f" candidate_uuid: "47532f24ee7641f994825b0a7d3f1682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79d20d5f21224c2091e691e77fce586f" is_pre_election: true
I20260925 10:08:36.310613 22173 raft_consensus.cc:2454] T 224e3ee58e6b43a0a98abfc3d1861e6f P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47532f24ee7641f994825b0a7d3f1682 in term 0.
I20260925 10:08:36.310640 22384 leader_election.cc:302] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47532f24ee7641f994825b0a7d3f1682, 50394402cef84a10b34012b8b108c29d; no voters:
I20260925 10:08:36.310729 22545 raft_consensus.cc:2781] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:36.310767 22545 raft_consensus.cc:482] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:36.310792 22545 raft_consensus.cc:3037] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:36.311576 22545 raft_consensus.cc:504] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.311873 22545 leader_election.cc:288] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682 [CANDIDATE]: Term 1 election: Requested vote from peers 79d20d5f21224c2091e691e77fce586f (127.14.41.65:42479), 50394402cef84a10b34012b8b108c29d (127.14.41.66:37351)
I20260925 10:08:36.312042 22173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224e3ee58e6b43a0a98abfc3d1861e6f" candidate_uuid: "47532f24ee7641f994825b0a7d3f1682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79d20d5f21224c2091e691e77fce586f"
I20260925 10:08:36.312065 22313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "224e3ee58e6b43a0a98abfc3d1861e6f" candidate_uuid: "47532f24ee7641f994825b0a7d3f1682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50394402cef84a10b34012b8b108c29d"
I20260925 10:08:36.312142 22313 raft_consensus.cc:3037] T 224e3ee58e6b43a0a98abfc3d1861e6f P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:36.312142 22173 raft_consensus.cc:3037] T 224e3ee58e6b43a0a98abfc3d1861e6f P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:36.313102 22173 raft_consensus.cc:2454] T 224e3ee58e6b43a0a98abfc3d1861e6f P 79d20d5f21224c2091e691e77fce586f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47532f24ee7641f994825b0a7d3f1682 in term 1.
I20260925 10:08:36.313107 22313 raft_consensus.cc:2454] T 224e3ee58e6b43a0a98abfc3d1861e6f P 50394402cef84a10b34012b8b108c29d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47532f24ee7641f994825b0a7d3f1682 in term 1.
I20260925 10:08:36.313321 22386 leader_election.cc:302] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47532f24ee7641f994825b0a7d3f1682, 79d20d5f21224c2091e691e77fce586f; no voters:
I20260925 10:08:36.313472 22545 raft_consensus.cc:2781] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:36.313607 22545 raft_consensus.cc:686] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682 [term 1 LEADER]: Becoming Leader. State: Replica: 47532f24ee7641f994825b0a7d3f1682, State: Running, Role: LEADER
I20260925 10:08:36.313663 22545 consensus_queue.cc:237] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.313970 22570 tablet_replica.cc:439] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682: TxnStatusTablet state changed. Reason: New leader 47532f24ee7641f994825b0a7d3f1682. Latest consensus state: current_term: 1 leader_uuid: "47532f24ee7641f994825b0a7d3f1682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } } }
I20260925 10:08:36.314072 22570 tablet_replica.cc:442] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:08:36.314229 22572 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 10:08:36.314384 22572 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 10:08:36.314395 22015 catalog_manager.cc:5495] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47532f24ee7641f994825b0a7d3f1682 (127.14.41.67). New cstate: current_term: 1 leader_uuid: "47532f24ee7641f994825b0a7d3f1682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } health_report { overall_health: HEALTHY } } }
I20260925 10:08:36.351665 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:36.349186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.14.41.67:46341)
0925 10:08:36.349240 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:36.349242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:36.349252 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:08:36.349357 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:36.349359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:36.349371 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:36.349435 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:36.349601 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.350090 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.350863 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.351084 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.351164 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:36.351345 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:36.351388 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:36.351458 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:36.351538 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 10:08:36.351563 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 10:08:36.351665 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:36.349096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:46341 (local address 127.0.0.1:53662)
0925 10:08:36.349184 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:36.349198 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:36.349254 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:36.349444 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:36.349444 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:36.349446 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:08:36.349527 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:36.349530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.350182 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.350184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:36.350773 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:36.350774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.351163 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.351164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:36.351237 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:36.351256 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:08:36.351479 (+ 223us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:08:36.351489 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 10:08:36.351535 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 10:08:36.351589 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 10:08:36.352918 22173 raft_consensus.cc:1261] T 224e3ee58e6b43a0a98abfc3d1861e6f P 79d20d5f21224c2091e691e77fce586f [term 1 FOLLOWER]: Refusing update from remote peer 47532f24ee7641f994825b0a7d3f1682: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:36.353014 22313 raft_consensus.cc:1261] T 224e3ee58e6b43a0a98abfc3d1861e6f P 50394402cef84a10b34012b8b108c29d [term 1 FOLLOWER]: Refusing update from remote peer 47532f24ee7641f994825b0a7d3f1682: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:36.353233 22570 consensus_queue.cc:1035] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:36.353343 22545 consensus_queue.cc:1035] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:36.353832 22533 tablet_replica.cc:439] T 224e3ee58e6b43a0a98abfc3d1861e6f P 79d20d5f21224c2091e691e77fce586f: TxnStatusTablet state changed. Reason: New leader 47532f24ee7641f994825b0a7d3f1682. Latest consensus state: current_term: 1 leader_uuid: "47532f24ee7641f994825b0a7d3f1682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } } }
I20260925 10:08:36.353929 22533 tablet_replica.cc:442] T 224e3ee58e6b43a0a98abfc3d1861e6f P 79d20d5f21224c2091e691e77fce586f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:08:36.353902 22535 tablet_replica.cc:439] T 224e3ee58e6b43a0a98abfc3d1861e6f P 50394402cef84a10b34012b8b108c29d: TxnStatusTablet state changed. Reason: New leader 47532f24ee7641f994825b0a7d3f1682. Latest consensus state: current_term: 1 leader_uuid: "47532f24ee7641f994825b0a7d3f1682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } } }
I20260925 10:08:36.353986 22535 tablet_replica.cc:442] T 224e3ee58e6b43a0a98abfc3d1861e6f P 50394402cef84a10b34012b8b108c29d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:08:36.354732 22545 tablet_replica.cc:439] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47532f24ee7641f994825b0a7d3f1682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } } }
I20260925 10:08:36.354830 22545 tablet_replica.cc:442] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:08:36.355054 22533 tablet_replica.cc:439] T 224e3ee58e6b43a0a98abfc3d1861e6f P 79d20d5f21224c2091e691e77fce586f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47532f24ee7641f994825b0a7d3f1682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } } }
I20260925 10:08:36.355139 22533 tablet_replica.cc:442] T 224e3ee58e6b43a0a98abfc3d1861e6f P 79d20d5f21224c2091e691e77fce586f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:08:36.355201 22575 mvcc.cc:205] Tried to move back new op lower bound from 7333195433379565568 to 7333195433221328896. Current Snapshot: MvccSnapshot[applied={T|T < 7333195433379565568 or (T in {7333195433379565568})}]
I20260925 10:08:36.355945 22545 tablet_replica.cc:439] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47532f24ee7641f994825b0a7d3f1682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } } }
I20260925 10:08:36.356031 22545 tablet_replica.cc:442] T 224e3ee58e6b43a0a98abfc3d1861e6f P 47532f24ee7641f994825b0a7d3f1682: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:08:36.356343 22535 tablet_replica.cc:439] T 224e3ee58e6b43a0a98abfc3d1861e6f P 50394402cef84a10b34012b8b108c29d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47532f24ee7641f994825b0a7d3f1682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } } }
I20260925 10:08:36.356423 22535 tablet_replica.cc:442] T 224e3ee58e6b43a0a98abfc3d1861e6f P 50394402cef84a10b34012b8b108c29d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:08:36.359741 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:36.356749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40036 (local address 127.14.41.66:37351)
0925 10:08:36.356941 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:36.356943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:36.356951 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:08:36.356986 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:36.356987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:36.357007 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:36.357090 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:36.357204 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.357956 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.358838 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.359168 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.359238 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:36.359399 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:36.359538 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:36.359617 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:36.359639 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 10:08:36.359661 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":56,"threads_started":1}
I20260925 10:08:36.364995 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:36.362306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:59989 (local address 127.14.41.67:46341)
0925 10:08:36.362359 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:36.362361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:36.362455 (+ 94us) server_negotiation.cc:407] Connection header received
0925 10:08:36.362513 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:36.362514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:36.362524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:36.362569 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:36.362686 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.363181 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.364317 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.364515 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.364574 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:36.364769 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:36.364794 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:36.364831 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:36.364920 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 10:08:36.364938 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 10:08:36.365041 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:36.362211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:46341 (local address 127.14.41.66:59989)
0925 10:08:36.362378 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:36.362390 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:36.362451 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:36.362574 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:36.362575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:36.362577 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:08:36.362630 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:36.362631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.363262 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.363263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:36.364235 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:36.364238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.364596 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.364597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:36.364670 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:36.364688 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:08:36.364856 (+ 168us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:08:36.364858 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:08:36.364907 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 10:08:36.364950 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20260925 10:08:36.367705 22453 raft_consensus.cc:1261] T 60f0d71a585d4fdc907c112445b3adb7 P 47532f24ee7641f994825b0a7d3f1682 [term 1 FOLLOWER]: Refusing update from remote peer 50394402cef84a10b34012b8b108c29d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:36.367728 22173 raft_consensus.cc:1261] T 60f0d71a585d4fdc907c112445b3adb7 P 79d20d5f21224c2091e691e77fce586f [term 1 FOLLOWER]: Refusing update from remote peer 50394402cef84a10b34012b8b108c29d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:36.368010 22558 consensus_queue.cc:1035] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:36.368167 22535 consensus_queue.cc:1035] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:36.383334 22593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:36.380387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:37351 (local address 127.14.41.67:41265)
0925 10:08:36.380601 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:36.380613 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:36.380678 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:36.380807 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:36.380807 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:36.380809 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:08:36.380858 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:36.380860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.381783 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.381784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:36.382447 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:36.382450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.382926 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.382927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:36.382975 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:36.382992 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:08:36.383186 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:08:36.383188 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:08:36.383223 (+ 35us) client_negotiation.cc:240] Negotiation successful
0925 10:08:36.383265 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":132,"threads_started":1}
I20260925 10:08:36.383342 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:36.380515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:41265 (local address 127.14.41.66:37351)
0925 10:08:36.380564 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:36.380567 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:36.380683 (+ 116us) server_negotiation.cc:407] Connection header received
0925 10:08:36.380720 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:36.380721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:36.380733 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:36.380797 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:36.380901 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.381679 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.382543 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.382819 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.382898 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:36.383081 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:36.383119 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:36.383176 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:36.383227 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 10:08:36.383250 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:08:36.391052 22015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:36.391178 22015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:36.391964 22015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de: Sending DeleteTablet for 3 replicas of tablet 60f0d71a585d4fdc907c112445b3adb7
I20260925 10:08:36.392264 22433 tablet_service.cc:1514] Processing DeleteTablet for tablet 60f0d71a585d4fdc907c112445b3adb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:36 UTC) from {username='oryx', principal='oryx/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:53644
I20260925 10:08:36.392320 22153 tablet_service.cc:1514] Processing DeleteTablet for tablet 60f0d71a585d4fdc907c112445b3adb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:36 UTC) from {username='oryx', principal='oryx/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:51572
I20260925 10:08:36.392385 22293 tablet_service.cc:1514] Processing DeleteTablet for tablet 60f0d71a585d4fdc907c112445b3adb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:36 UTC) from {username='oryx', principal='oryx/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:40034
I20260925 10:08:36.392601 22598 tablet_replica.cc:330] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d: stopping tablet replica
I20260925 10:08:36.392661 22598 raft_consensus.cc:2229] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:08:36.392766 22598 raft_consensus.cc:2258] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:36.393049 22598 ts_tablet_manager.cc:1905] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:36.394961 22598 ts_tablet_manager.cc:1918] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:08:36.395015 22598 log.cc:1196] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/60f0d71a585d4fdc907c112445b3adb7
I20260925 10:08:36.395248 22598 ts_tablet_manager.cc:1939] T 60f0d71a585d4fdc907c112445b3adb7 P 50394402cef84a10b34012b8b108c29d: Deleting consensus metadata
I20260925 10:08:36.395680 22000 catalog_manager.cc:4841] TS 50394402cef84a10b34012b8b108c29d (127.14.41.66:37351): tablet 60f0d71a585d4fdc907c112445b3adb7 (table test-table [id=3d8b32b7fb0e44cb9000aee26642c17b]) successfully deleted
I20260925 10:08:36.395697 22596 tablet_replica.cc:330] T 60f0d71a585d4fdc907c112445b3adb7 P 47532f24ee7641f994825b0a7d3f1682: stopping tablet replica
I20260925 10:08:36.395834 22596 raft_consensus.cc:2229] T 60f0d71a585d4fdc907c112445b3adb7 P 47532f24ee7641f994825b0a7d3f1682 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:36.395965 22596 raft_consensus.cc:2258] T 60f0d71a585d4fdc907c112445b3adb7 P 47532f24ee7641f994825b0a7d3f1682 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:36.396294 22596 ts_tablet_manager.cc:1905] T 60f0d71a585d4fdc907c112445b3adb7 P 47532f24ee7641f994825b0a7d3f1682: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:36.398005 22596 ts_tablet_manager.cc:1918] T 60f0d71a585d4fdc907c112445b3adb7 P 47532f24ee7641f994825b0a7d3f1682: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:08:36.398123 22596 log.cc:1196] T 60f0d71a585d4fdc907c112445b3adb7 P 47532f24ee7641f994825b0a7d3f1682: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/60f0d71a585d4fdc907c112445b3adb7
I20260925 10:08:36.398396 22596 ts_tablet_manager.cc:1939] T 60f0d71a585d4fdc907c112445b3adb7 P 47532f24ee7641f994825b0a7d3f1682: Deleting consensus metadata
I20260925 10:08:36.398847 22001 catalog_manager.cc:4841] TS 47532f24ee7641f994825b0a7d3f1682 (127.14.41.67:46341): tablet 60f0d71a585d4fdc907c112445b3adb7 (table test-table [id=3d8b32b7fb0e44cb9000aee26642c17b]) successfully deleted
I20260925 10:08:36.400098 22597 tablet_replica.cc:330] T 60f0d71a585d4fdc907c112445b3adb7 P 79d20d5f21224c2091e691e77fce586f: stopping tablet replica
I20260925 10:08:36.400192 22597 raft_consensus.cc:2229] T 60f0d71a585d4fdc907c112445b3adb7 P 79d20d5f21224c2091e691e77fce586f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:36.400242 22597 raft_consensus.cc:2258] T 60f0d71a585d4fdc907c112445b3adb7 P 79d20d5f21224c2091e691e77fce586f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:36.400484 22597 ts_tablet_manager.cc:1905] T 60f0d71a585d4fdc907c112445b3adb7 P 79d20d5f21224c2091e691e77fce586f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:36.402472 22597 ts_tablet_manager.cc:1918] T 60f0d71a585d4fdc907c112445b3adb7 P 79d20d5f21224c2091e691e77fce586f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:08:36.402547 22597 log.cc:1196] T 60f0d71a585d4fdc907c112445b3adb7 P 79d20d5f21224c2091e691e77fce586f: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-0/wal/wals/60f0d71a585d4fdc907c112445b3adb7
I20260925 10:08:36.402783 22597 ts_tablet_manager.cc:1939] T 60f0d71a585d4fdc907c112445b3adb7 P 79d20d5f21224c2091e691e77fce586f: Deleting consensus metadata
I20260925 10:08:36.403519 22002 catalog_manager.cc:4841] TS 79d20d5f21224c2091e691e77fce586f (127.14.41.65:42479): tablet 60f0d71a585d4fdc907c112445b3adb7 (table test-table [id=3d8b32b7fb0e44cb9000aee26642c17b]) successfully deleted
Sep 25 10:08:36 dist-test-slave-s6tm krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:08:36 dist-test-slave-s6tm krb5kdc[21971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:36 dist-test-slave-s6tm krb5kdc[21971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:08:36.422199 22566 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:36.416467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.14.41.126:34013)
0925 10:08:36.416523 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:36.416525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:36.416704 (+ 179us) server_negotiation.cc:407] Connection header received
0925 10:08:36.416789 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:36.416790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:36.416813 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:36.416878 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:36.417564 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.418074 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.418846 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.419064 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.419150 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:36.422120 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43876: BlockingRecv error: recv got EOF from 127.0.0.1:43876 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 10:08:36 dist-test-slave-s6tm krb5kdc[21971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM
I20260925 10:08:36.430961 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:36.423099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43882 (local address 127.14.41.126:34013)
0925 10:08:36.423138 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:36.423140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:36.423391 (+ 251us) server_negotiation.cc:407] Connection header received
0925 10:08:36.423471 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:36.423471 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:36.423482 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:36.423551 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:36.424234 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.424843 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.425565 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.425775 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.425852 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:36.428123 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:36.428124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:36.428125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:36.428133 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:36.429625 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:36.429997 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:36.429998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:36.429999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:36.430059 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:36.430321 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:36.430323 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:36.430323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:36.430448 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:36.430560 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:36.430728 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 10:08:36.430813 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:08:36.432613 22015 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:08:36.432819 22015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:08:36.434175 22433 tablet_service.cc:1467] Processing CreateTablet for tablet 953965681cbc4b9dbdb6bb7e7bb38af2 (DEFAULT_TABLE table=test-table [id=b0d33bb278364f47941839a9b2f5c851]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:36.434175 22293 tablet_service.cc:1467] Processing CreateTablet for tablet 953965681cbc4b9dbdb6bb7e7bb38af2 (DEFAULT_TABLE table=test-table [id=b0d33bb278364f47941839a9b2f5c851]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:36.434314 22433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953965681cbc4b9dbdb6bb7e7bb38af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:36.434312 22293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953965681cbc4b9dbdb6bb7e7bb38af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:36.435675 22529 tablet_bootstrap.cc:492] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 50394402cef84a10b34012b8b108c29d: Bootstrap starting.
I20260925 10:08:36.436158 22529 tablet_bootstrap.cc:654] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 50394402cef84a10b34012b8b108c29d: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:36.435956 22528 tablet_bootstrap.cc:492] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 47532f24ee7641f994825b0a7d3f1682: Bootstrap starting.
I20260925 10:08:36.436544 22528 tablet_bootstrap.cc:654] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 47532f24ee7641f994825b0a7d3f1682: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:36.437148 22528 tablet_bootstrap.cc:492] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 47532f24ee7641f994825b0a7d3f1682: No bootstrap required, opened a new log
I20260925 10:08:36.437196 22528 ts_tablet_manager.cc:1397] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 47532f24ee7641f994825b0a7d3f1682: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:36.437295 22528 raft_consensus.cc:348] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.437358 22528 raft_consensus.cc:374] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:36.437383 22528 raft_consensus.cc:729] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47532f24ee7641f994825b0a7d3f1682, State: Initialized, Role: FOLLOWER
I20260925 10:08:36.437430 22528 consensus_queue.cc:260] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 47532f24ee7641f994825b0a7d3f1682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.437537 22528 ts_tablet_manager.cc:1428] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 47532f24ee7641f994825b0a7d3f1682: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:36.437834 22153 tablet_service.cc:1467] Processing CreateTablet for tablet 953965681cbc4b9dbdb6bb7e7bb38af2 (DEFAULT_TABLE table=test-table [id=b0d33bb278364f47941839a9b2f5c851]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:36.437947 22153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953965681cbc4b9dbdb6bb7e7bb38af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:36.438846 22527 tablet_bootstrap.cc:492] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f: Bootstrap starting.
I20260925 10:08:36.439281 22527 tablet_bootstrap.cc:654] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:36.439960 22529 tablet_bootstrap.cc:492] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 50394402cef84a10b34012b8b108c29d: No bootstrap required, opened a new log
I20260925 10:08:36.440016 22529 ts_tablet_manager.cc:1397] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 50394402cef84a10b34012b8b108c29d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260925 10:08:36.440183 22529 raft_consensus.cc:348] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.440271 22529 raft_consensus.cc:374] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:36.440272 22527 tablet_bootstrap.cc:492] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f: No bootstrap required, opened a new log
I20260925 10:08:36.440297 22529 raft_consensus.cc:729] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50394402cef84a10b34012b8b108c29d, State: Initialized, Role: FOLLOWER
I20260925 10:08:36.440311 22527 ts_tablet_manager.cc:1397] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:36.440344 22529 consensus_queue.cc:260] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 50394402cef84a10b34012b8b108c29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.440466 22529 ts_tablet_manager.cc:1428] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 50394402cef84a10b34012b8b108c29d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:36.440460 22527 raft_consensus.cc:348] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.440532 22527 raft_consensus.cc:374] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:36.440558 22527 raft_consensus.cc:729] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d20d5f21224c2091e691e77fce586f, State: Initialized, Role: FOLLOWER
I20260925 10:08:36.440609 22527 consensus_queue.cc:260] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.440688 22527 ts_tablet_manager.cc:1428] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:36.510064 22533 raft_consensus.cc:482] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:36.510190 22533 raft_consensus.cc:504] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.510550 22533 leader_election.cc:288] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50394402cef84a10b34012b8b108c29d (127.14.41.66:37351), 47532f24ee7641f994825b0a7d3f1682 (127.14.41.67:46341)
I20260925 10:08:36.514112 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:36.510854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:44853 (local address 127.14.41.66:37351)
0925 10:08:36.510910 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:36.510913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:36.510971 (+ 58us) server_negotiation.cc:407] Connection header received
0925 10:08:36.511086 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:36.511087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:36.511098 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:36.511130 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:36.511375 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.511871 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.513358 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.513729 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.513816 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:36.513827 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:36.514000 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 10:08:36.514040 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:08:36.514544 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:36.510695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:37351 (local address 127.14.41.65:44853)
0925 10:08:36.510905 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:36.510916 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:36.511003 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:36.511214 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:36.511214 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:36.511220 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:36.511294 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:36.511296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.512065 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.512067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:36.513283 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:36.513286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.513816 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.513817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:36.513901 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:36.513908 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:08:36.513989 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 10:08:36.514034 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":68,"threads_started":1}
I20260925 10:08:36.514966 22313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953965681cbc4b9dbdb6bb7e7bb38af2" candidate_uuid: "79d20d5f21224c2091e691e77fce586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50394402cef84a10b34012b8b108c29d" is_pre_election: true
I20260925 10:08:36.515070 22313 raft_consensus.cc:2454] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79d20d5f21224c2091e691e77fce586f in term 0.
I20260925 10:08:36.515282 22104 leader_election.cc:302] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50394402cef84a10b34012b8b108c29d, 79d20d5f21224c2091e691e77fce586f; no voters:
I20260925 10:08:36.515384 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:36.511042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:39221 (local address 127.14.41.67:46341)
0925 10:08:36.511086 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:36.511088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:36.511238 (+ 150us) server_negotiation.cc:407] Connection header received
0925 10:08:36.511272 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:36.511274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:36.511283 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:36.511357 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:36.512211 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.512931 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.514333 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.514914 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.514983 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:36.514993 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:36.515266 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 10:08:36.515317 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:08:36.515383 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:36.510803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:46341 (local address 127.14.41.65:39221)
0925 10:08:36.511158 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:36.511168 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:36.511223 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:36.512030 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:36.512033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:36.512037 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:36.512120 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:36.512122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.513030 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.513031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:36.514242 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:36.514245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.515060 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.515062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:36.515148 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:36.515154 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:08:36.515251 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 10:08:36.515295 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":118,"threads_started":1}
I20260925 10:08:36.515439 22533 raft_consensus.cc:2781] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:36.515472 22533 raft_consensus.cc:482] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:36.515496 22533 raft_consensus.cc:3037] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:36.515570 22453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953965681cbc4b9dbdb6bb7e7bb38af2" candidate_uuid: "79d20d5f21224c2091e691e77fce586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47532f24ee7641f994825b0a7d3f1682" is_pre_election: true
I20260925 10:08:36.515647 22453 raft_consensus.cc:2454] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79d20d5f21224c2091e691e77fce586f in term 0.
I20260925 10:08:36.516518 22533 raft_consensus.cc:504] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.516680 22533 leader_election.cc:288] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [CANDIDATE]: Term 1 election: Requested vote from peers 50394402cef84a10b34012b8b108c29d (127.14.41.66:37351), 47532f24ee7641f994825b0a7d3f1682 (127.14.41.67:46341)
I20260925 10:08:36.516853 22313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953965681cbc4b9dbdb6bb7e7bb38af2" candidate_uuid: "79d20d5f21224c2091e691e77fce586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50394402cef84a10b34012b8b108c29d"
I20260925 10:08:36.516933 22313 raft_consensus.cc:3037] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:36.516924 22453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953965681cbc4b9dbdb6bb7e7bb38af2" candidate_uuid: "79d20d5f21224c2091e691e77fce586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47532f24ee7641f994825b0a7d3f1682"
I20260925 10:08:36.516984 22453 raft_consensus.cc:3037] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:36.517733 22453 raft_consensus.cc:2454] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 47532f24ee7641f994825b0a7d3f1682 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79d20d5f21224c2091e691e77fce586f in term 1.
I20260925 10:08:36.517738 22313 raft_consensus.cc:2454] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 50394402cef84a10b34012b8b108c29d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79d20d5f21224c2091e691e77fce586f in term 1.
I20260925 10:08:36.517906 22104 leader_election.cc:302] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50394402cef84a10b34012b8b108c29d, 79d20d5f21224c2091e691e77fce586f; no voters:
I20260925 10:08:36.517998 22533 raft_consensus.cc:2781] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:36.518182 22533 raft_consensus.cc:686] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [term 1 LEADER]: Becoming Leader. State: Replica: 79d20d5f21224c2091e691e77fce586f, State: Running, Role: LEADER
I20260925 10:08:36.518255 22533 consensus_queue.cc:237] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.519091 22014 catalog_manager.cc:5495] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f reported cstate change: term changed from 0 to 1, leader changed from <none> to 79d20d5f21224c2091e691e77fce586f (127.14.41.65). New cstate: current_term: 1 leader_uuid: "79d20d5f21224c2091e691e77fce586f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } health_report { overall_health: UNKNOWN } } }
I20260925 10:08:36.534368 22619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:36.531587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51580 (local address 127.14.41.65:42479)
0925 10:08:36.531776 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:36.531778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:36.531784 (+ 6us) server_negotiation.cc:407] Connection header received
0925 10:08:36.531822 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:36.531822 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:36.531841 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:36.532029 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:36.532183 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.532724 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.533513 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.533741 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.533813 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:36.533991 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:36.534103 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:36.534191 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:36.534229 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 10:08:36.534251 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":113,"threads_started":1}
I20260925 10:08:36.535470 22313 raft_consensus.cc:1261] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 50394402cef84a10b34012b8b108c29d [term 1 FOLLOWER]: Refusing update from remote peer 79d20d5f21224c2091e691e77fce586f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:36.535460 22453 raft_consensus.cc:1261] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 47532f24ee7641f994825b0a7d3f1682 [term 1 FOLLOWER]: Refusing update from remote peer 79d20d5f21224c2091e691e77fce586f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:36.535768 22533 consensus_queue.cc:1035] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:36.535984 22616 consensus_queue.cc:1035] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:36.539758 22016 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:36.539868 22016 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:36.540532 22016 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de: Sending DeleteTablet for 3 replicas of tablet 953965681cbc4b9dbdb6bb7e7bb38af2
I20260925 10:08:36.540777 22293 tablet_service.cc:1514] Processing DeleteTablet for tablet 953965681cbc4b9dbdb6bb7e7bb38af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:36 UTC) from {username='oryx', principal='oryx/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:40034
I20260925 10:08:36.540813 22433 tablet_service.cc:1514] Processing DeleteTablet for tablet 953965681cbc4b9dbdb6bb7e7bb38af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:36 UTC) from {username='oryx', principal='oryx/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:53644
I20260925 10:08:36.540898 22596 tablet_replica.cc:330] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 47532f24ee7641f994825b0a7d3f1682: stopping tablet replica
I20260925 10:08:36.540894 22598 tablet_replica.cc:330] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 50394402cef84a10b34012b8b108c29d: stopping tablet replica
I20260925 10:08:36.540953 22596 raft_consensus.cc:2229] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 47532f24ee7641f994825b0a7d3f1682 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:36.540961 22598 raft_consensus.cc:2229] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 50394402cef84a10b34012b8b108c29d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:36.540997 22596 raft_consensus.cc:2258] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 47532f24ee7641f994825b0a7d3f1682 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:36.541004 22598 raft_consensus.cc:2258] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 50394402cef84a10b34012b8b108c29d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:36.541113 22153 tablet_service.cc:1514] Processing DeleteTablet for tablet 953965681cbc4b9dbdb6bb7e7bb38af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:36 UTC) from {username='oryx', principal='oryx/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:51572
I20260925 10:08:36.541208 22597 tablet_replica.cc:330] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f: stopping tablet replica
I20260925 10:08:36.541275 22597 raft_consensus.cc:2229] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:08:36.541271 22016 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 10:08:36.541365 22597 raft_consensus.cc:2258] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 10:08:36.541414 22016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:08:36.541677 22598 ts_tablet_manager.cc:1905] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 50394402cef84a10b34012b8b108c29d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:36.542667 22153 tablet_service.cc:1467] Processing CreateTablet for tablet 5aff4d81fc8249709c6a9f15460b9270 (DEFAULT_TABLE table=test-table [id=0e8a60b661e04973b094f235639510ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:36.542714 22293 tablet_service.cc:1467] Processing CreateTablet for tablet 5aff4d81fc8249709c6a9f15460b9270 (DEFAULT_TABLE table=test-table [id=0e8a60b661e04973b094f235639510ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:36.542793 22153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aff4d81fc8249709c6a9f15460b9270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:36.542855 22293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aff4d81fc8249709c6a9f15460b9270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:36.542980 22433 tablet_service.cc:1467] Processing CreateTablet for tablet 5aff4d81fc8249709c6a9f15460b9270 (DEFAULT_TABLE table=test-table [id=0e8a60b661e04973b094f235639510ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:36.543078 22433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aff4d81fc8249709c6a9f15460b9270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:36.543537 22598 ts_tablet_manager.cc:1918] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 50394402cef84a10b34012b8b108c29d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:08:36.543596 22598 log.cc:1196] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 50394402cef84a10b34012b8b108c29d: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/953965681cbc4b9dbdb6bb7e7bb38af2
I20260925 10:08:36.543815 22598 ts_tablet_manager.cc:1939] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 50394402cef84a10b34012b8b108c29d: Deleting consensus metadata
I20260925 10:08:36.544207 22527 tablet_bootstrap.cc:492] T 5aff4d81fc8249709c6a9f15460b9270 P 79d20d5f21224c2091e691e77fce586f: Bootstrap starting.
I20260925 10:08:36.544562 22528 tablet_bootstrap.cc:492] T 5aff4d81fc8249709c6a9f15460b9270 P 47532f24ee7641f994825b0a7d3f1682: Bootstrap starting.
I20260925 10:08:36.544625 22527 tablet_bootstrap.cc:654] T 5aff4d81fc8249709c6a9f15460b9270 P 79d20d5f21224c2091e691e77fce586f: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:36.544692 22529 tablet_bootstrap.cc:492] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d: Bootstrap starting.
I20260925 10:08:36.544891 22528 tablet_bootstrap.cc:654] T 5aff4d81fc8249709c6a9f15460b9270 P 47532f24ee7641f994825b0a7d3f1682: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:36.544997 22529 tablet_bootstrap.cc:654] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:36.545004 22000 catalog_manager.cc:4841] TS 50394402cef84a10b34012b8b108c29d (127.14.41.66:37351): tablet 953965681cbc4b9dbdb6bb7e7bb38af2 (table test-table [id=b0d33bb278364f47941839a9b2f5c851]) successfully deleted
I20260925 10:08:36.545442 22596 ts_tablet_manager.cc:1905] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 47532f24ee7641f994825b0a7d3f1682: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:36.545717 22527 tablet_bootstrap.cc:492] T 5aff4d81fc8249709c6a9f15460b9270 P 79d20d5f21224c2091e691e77fce586f: No bootstrap required, opened a new log
I20260925 10:08:36.545766 22527 ts_tablet_manager.cc:1397] T 5aff4d81fc8249709c6a9f15460b9270 P 79d20d5f21224c2091e691e77fce586f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:08:36.545933 22527 raft_consensus.cc:348] T 5aff4d81fc8249709c6a9f15460b9270 P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.546010 22527 raft_consensus.cc:374] T 5aff4d81fc8249709c6a9f15460b9270 P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:36.546036 22527 raft_consensus.cc:729] T 5aff4d81fc8249709c6a9f15460b9270 P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d20d5f21224c2091e691e77fce586f, State: Initialized, Role: FOLLOWER
I20260925 10:08:36.546085 22527 consensus_queue.cc:260] T 5aff4d81fc8249709c6a9f15460b9270 P 79d20d5f21224c2091e691e77fce586f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.546193 22527 ts_tablet_manager.cc:1428] T 5aff4d81fc8249709c6a9f15460b9270 P 79d20d5f21224c2091e691e77fce586f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:36.545440 22528 tablet_bootstrap.cc:492] T 5aff4d81fc8249709c6a9f15460b9270 P 47532f24ee7641f994825b0a7d3f1682: No bootstrap required, opened a new log
I20260925 10:08:36.546355 22597 ts_tablet_manager.cc:1905] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:36.546384 22528 ts_tablet_manager.cc:1397] T 5aff4d81fc8249709c6a9f15460b9270 P 47532f24ee7641f994825b0a7d3f1682: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:08:36.546483 22528 raft_consensus.cc:348] T 5aff4d81fc8249709c6a9f15460b9270 P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.546538 22528 raft_consensus.cc:374] T 5aff4d81fc8249709c6a9f15460b9270 P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:36.546555 22528 raft_consensus.cc:729] T 5aff4d81fc8249709c6a9f15460b9270 P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47532f24ee7641f994825b0a7d3f1682, State: Initialized, Role: FOLLOWER
I20260925 10:08:36.546594 22528 consensus_queue.cc:260] T 5aff4d81fc8249709c6a9f15460b9270 P 47532f24ee7641f994825b0a7d3f1682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.546674 22528 ts_tablet_manager.cc:1428] T 5aff4d81fc8249709c6a9f15460b9270 P 47532f24ee7641f994825b0a7d3f1682: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:36.547083 22596 ts_tablet_manager.cc:1918] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 47532f24ee7641f994825b0a7d3f1682: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:08:36.547142 22596 log.cc:1196] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 47532f24ee7641f994825b0a7d3f1682: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/953965681cbc4b9dbdb6bb7e7bb38af2
I20260925 10:08:36.547371 22596 ts_tablet_manager.cc:1939] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 47532f24ee7641f994825b0a7d3f1682: Deleting consensus metadata
I20260925 10:08:36.547780 22001 catalog_manager.cc:4841] TS 47532f24ee7641f994825b0a7d3f1682 (127.14.41.67:46341): tablet 953965681cbc4b9dbdb6bb7e7bb38af2 (table test-table [id=b0d33bb278364f47941839a9b2f5c851]) successfully deleted
I20260925 10:08:36.547878 22597 ts_tablet_manager.cc:1918] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:08:36.547953 22597 log.cc:1196] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-0/wal/wals/953965681cbc4b9dbdb6bb7e7bb38af2
I20260925 10:08:36.548159 22597 ts_tablet_manager.cc:1939] T 953965681cbc4b9dbdb6bb7e7bb38af2 P 79d20d5f21224c2091e691e77fce586f: Deleting consensus metadata
I20260925 10:08:36.548431 22529 tablet_bootstrap.cc:492] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d: No bootstrap required, opened a new log
I20260925 10:08:36.548489 22529 ts_tablet_manager.cc:1397] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260925 10:08:36.548599 22002 catalog_manager.cc:4841] TS 79d20d5f21224c2091e691e77fce586f (127.14.41.65:42479): tablet 953965681cbc4b9dbdb6bb7e7bb38af2 (table test-table [id=b0d33bb278364f47941839a9b2f5c851]) successfully deleted
I20260925 10:08:36.548601 22529 raft_consensus.cc:348] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.548651 22529 raft_consensus.cc:374] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:36.548664 22529 raft_consensus.cc:729] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50394402cef84a10b34012b8b108c29d, State: Initialized, Role: FOLLOWER
I20260925 10:08:36.548727 22529 consensus_queue.cc:260] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.548823 22529 ts_tablet_manager.cc:1428] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:36.762415 22558 raft_consensus.cc:482] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:36.762532 22558 raft_consensus.cc:504] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.762722 22558 leader_election.cc:288] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79d20d5f21224c2091e691e77fce586f (127.14.41.65:42479), 47532f24ee7641f994825b0a7d3f1682 (127.14.41.67:46341)
I20260925 10:08:36.762993 22173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aff4d81fc8249709c6a9f15460b9270" candidate_uuid: "50394402cef84a10b34012b8b108c29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79d20d5f21224c2091e691e77fce586f" is_pre_election: true
I20260925 10:08:36.763015 22453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aff4d81fc8249709c6a9f15460b9270" candidate_uuid: "50394402cef84a10b34012b8b108c29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47532f24ee7641f994825b0a7d3f1682" is_pre_election: true
I20260925 10:08:36.763108 22453 raft_consensus.cc:2454] T 5aff4d81fc8249709c6a9f15460b9270 P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50394402cef84a10b34012b8b108c29d in term 0.
I20260925 10:08:36.763108 22173 raft_consensus.cc:2454] T 5aff4d81fc8249709c6a9f15460b9270 P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50394402cef84a10b34012b8b108c29d in term 0.
I20260925 10:08:36.763301 22246 leader_election.cc:302] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50394402cef84a10b34012b8b108c29d, 79d20d5f21224c2091e691e77fce586f; no voters:
I20260925 10:08:36.763460 22558 raft_consensus.cc:2781] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:36.763548 22558 raft_consensus.cc:482] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:36.763581 22558 raft_consensus.cc:3037] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:36.764482 22558 raft_consensus.cc:504] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.764684 22558 leader_election.cc:288] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [CANDIDATE]: Term 1 election: Requested vote from peers 79d20d5f21224c2091e691e77fce586f (127.14.41.65:42479), 47532f24ee7641f994825b0a7d3f1682 (127.14.41.67:46341)
I20260925 10:08:36.764882 22453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aff4d81fc8249709c6a9f15460b9270" candidate_uuid: "50394402cef84a10b34012b8b108c29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47532f24ee7641f994825b0a7d3f1682"
I20260925 10:08:36.764979 22453 raft_consensus.cc:3037] T 5aff4d81fc8249709c6a9f15460b9270 P 47532f24ee7641f994825b0a7d3f1682 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:36.764989 22173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aff4d81fc8249709c6a9f15460b9270" candidate_uuid: "50394402cef84a10b34012b8b108c29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79d20d5f21224c2091e691e77fce586f"
I20260925 10:08:36.765126 22173 raft_consensus.cc:3037] T 5aff4d81fc8249709c6a9f15460b9270 P 79d20d5f21224c2091e691e77fce586f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:36.765729 22453 raft_consensus.cc:2454] T 5aff4d81fc8249709c6a9f15460b9270 P 47532f24ee7641f994825b0a7d3f1682 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50394402cef84a10b34012b8b108c29d in term 1.
I20260925 10:08:36.765780 22173 raft_consensus.cc:2454] T 5aff4d81fc8249709c6a9f15460b9270 P 79d20d5f21224c2091e691e77fce586f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50394402cef84a10b34012b8b108c29d in term 1.
I20260925 10:08:36.765992 22246 leader_election.cc:302] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50394402cef84a10b34012b8b108c29d, 79d20d5f21224c2091e691e77fce586f; no voters:
I20260925 10:08:36.766114 22558 raft_consensus.cc:2781] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:36.766161 22558 raft_consensus.cc:686] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [term 1 LEADER]: Becoming Leader. State: Replica: 50394402cef84a10b34012b8b108c29d, State: Running, Role: LEADER
I20260925 10:08:36.766230 22558 consensus_queue.cc:237] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } }
I20260925 10:08:36.766937 22016 catalog_manager.cc:5495] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d reported cstate change: term changed from 0 to 1, leader changed from <none> to 50394402cef84a10b34012b8b108c29d (127.14.41.66). New cstate: current_term: 1 leader_uuid: "50394402cef84a10b34012b8b108c29d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50394402cef84a10b34012b8b108c29d" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 37351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 } health_report { overall_health: UNKNOWN } } }
I20260925 10:08:36.778259 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:36.775685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40044 (local address 127.14.41.66:37351)
0925 10:08:36.775743 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:36.775745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:36.775754 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:08:36.775776 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:36.775777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:36.775798 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:36.775842 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:36.776031 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.776501 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.777365 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:36.777576 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:36.777635 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:36.777938 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:36.777985 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:36.778052 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:36.778119 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 10:08:36.778148 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 10:08:36.781139 22173 raft_consensus.cc:1261] T 5aff4d81fc8249709c6a9f15460b9270 P 79d20d5f21224c2091e691e77fce586f [term 1 FOLLOWER]: Refusing update from remote peer 50394402cef84a10b34012b8b108c29d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:36.781123 22453 raft_consensus.cc:1261] T 5aff4d81fc8249709c6a9f15460b9270 P 47532f24ee7641f994825b0a7d3f1682 [term 1 FOLLOWER]: Refusing update from remote peer 50394402cef84a10b34012b8b108c29d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:36.781385 22535 consensus_queue.cc:1035] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "47532f24ee7641f994825b0a7d3f1682" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:36.781548 22558 consensus_queue.cc:1035] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "79d20d5f21224c2091e691e77fce586f" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:36.792507 22014 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:36.792627 22014 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:36.793597 22014 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bd1639b766e409eac6dcf2e872f66de: Sending DeleteTablet for 3 replicas of tablet 5aff4d81fc8249709c6a9f15460b9270
I20260925 10:08:36.793913 22293 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aff4d81fc8249709c6a9f15460b9270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:36 UTC) from {username='oryx', principal='oryx/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:40034
I20260925 10:08:36.794056 22598 tablet_replica.cc:330] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d: stopping tablet replica
I20260925 10:08:36.794132 22598 raft_consensus.cc:2229] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:08:36.794112 22153 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aff4d81fc8249709c6a9f15460b9270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:36 UTC) from {username='oryx', principal='oryx/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:51572
I20260925 10:08:36.794203 22598 raft_consensus.cc:2258] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:36.794348 22597 tablet_replica.cc:330] T 5aff4d81fc8249709c6a9f15460b9270 P 79d20d5f21224c2091e691e77fce586f: stopping tablet replica
I20260925 10:08:36.794339 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 22088
I20260925 10:08:36.794400 22597 raft_consensus.cc:2229] T 5aff4d81fc8249709c6a9f15460b9270 P 79d20d5f21224c2091e691e77fce586f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:36.794447 22597 raft_consensus.cc:2258] T 5aff4d81fc8249709c6a9f15460b9270 P 79d20d5f21224c2091e691e77fce586f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:36.795403 22598 ts_tablet_manager.cc:1905] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:36.796881 22433 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aff4d81fc8249709c6a9f15460b9270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:36 UTC) from {username='oryx', principal='oryx/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:53644
I20260925 10:08:36.796998 22596 tablet_replica.cc:330] T 5aff4d81fc8249709c6a9f15460b9270 P 47532f24ee7641f994825b0a7d3f1682: stopping tablet replica
I20260925 10:08:36.797078 22596 raft_consensus.cc:2229] T 5aff4d81fc8249709c6a9f15460b9270 P 47532f24ee7641f994825b0a7d3f1682 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:36.797142 22596 raft_consensus.cc:2258] T 5aff4d81fc8249709c6a9f15460b9270 P 47532f24ee7641f994825b0a7d3f1682 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:36.797273 22598 ts_tablet_manager.cc:1918] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:08:36.797338 22598 log.cc:1196] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/5aff4d81fc8249709c6a9f15460b9270
I20260925 10:08:36.797538 22598 ts_tablet_manager.cc:1939] T 5aff4d81fc8249709c6a9f15460b9270 P 50394402cef84a10b34012b8b108c29d: Deleting consensus metadata
I20260925 10:08:36.798013 22596 ts_tablet_manager.cc:1905] T 5aff4d81fc8249709c6a9f15460b9270 P 47532f24ee7641f994825b0a7d3f1682: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:36.799160 22596 ts_tablet_manager.cc:1918] T 5aff4d81fc8249709c6a9f15460b9270 P 47532f24ee7641f994825b0a7d3f1682: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:08:36.799209 22596 log.cc:1196] T 5aff4d81fc8249709c6a9f15460b9270 P 47532f24ee7641f994825b0a7d3f1682: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/5aff4d81fc8249709c6a9f15460b9270
I20260925 10:08:36.799417 22596 ts_tablet_manager.cc:1939] T 5aff4d81fc8249709c6a9f15460b9270 P 47532f24ee7641f994825b0a7d3f1682: Deleting consensus metadata
I20260925 10:08:36.799767 22001 catalog_manager.cc:4841] TS 47532f24ee7641f994825b0a7d3f1682 (127.14.41.67:46341): tablet 5aff4d81fc8249709c6a9f15460b9270 (table test-table [id=0e8a60b661e04973b094f235639510ef]) successfully deleted
I20260925 10:08:36.800064 22000 catalog_manager.cc:4841] TS 50394402cef84a10b34012b8b108c29d (127.14.41.66:37351): tablet 5aff4d81fc8249709c6a9f15460b9270 (table test-table [id=0e8a60b661e04973b094f235639510ef]) successfully deleted
W20260925 10:08:36.801391 22002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.41.65:42479 (error 108)
I20260925 10:08:36.802007 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 22228
I20260925 10:08:36.802260 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:36.802021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:42479 (local address 127.0.0.1:51586)
0925 10:08:36.802166 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:36.802214 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.65:42479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
W20260925 10:08:36.803188 22002 catalog_manager.cc:4584] TS 79d20d5f21224c2091e691e77fce586f (127.14.41.65:42479): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5aff4d81fc8249709c6a9f15460b9270: Network error: Client connection negotiation failed: client connection to 127.14.41.65:42479: connect: Connection refused (error 111)
I20260925 10:08:36.808900 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 22368
I20260925 10:08:36.815256 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 21984
2026-09-25T10:08:36Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3424 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:08:36 dist-test-slave-s6tm krb5kdc[22635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:08:36 dist-test-slave-s6tm krb5kdc[22635](info): set up 2 sockets
Sep 25 10:08:36 dist-test-slave-s6tm krb5kdc[22635](info): commencing operation
krb5kdc: starting...
W20260925 10:08:37.869937 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:08:37 dist-test-slave-s6tm krb5kdc[22635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:08:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:08:37.999714 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:35093
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:35093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:38.084815 22648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:38.085047 22648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:38.085073 22648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:38.086326 22648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:38.086382 22648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:38.086403 22648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:38.086417 22648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:38.086429 22648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:08:38.088079 22648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:35093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:35093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:38.088351 22648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:38.088626 22648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:38.093977 22656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:38.093977 22653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:38.094019 22654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:38.094389 22648 server_base.cc:1034] running on GCE node
I20260925 10:08:38.094695 22648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:38.095055 22648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:38.096232 22648 hybrid_clock.cc:648] HybridClock initialized: now 1790330918096203 us; error 49 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T10:08:38Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1290 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:08:38 dist-test-slave-s6tm krb5kdc[22661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:08:38 dist-test-slave-s6tm krb5kdc[22661](info): set up 2 sockets
Sep 25 10:08:38 dist-test-slave-s6tm krb5kdc[22661](info): commencing operation
krb5kdc: starting...
W20260925 10:08:39.173982 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:08:39 dist-test-slave-s6tm krb5kdc[22661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:08:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:08:39.303714 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:46043
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:44837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:46043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:39.386675 22674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:39.386924 22674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:39.386961 22674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:39.388252 22674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:39.388315 22674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:39.388339 22674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:39.388368 22674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:39.388398 22674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:08:39.389899 22674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:44837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:46043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:46043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:39.390192 22674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:39.390435 22674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:39.395735 22680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:39.395735 22682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:39.396091 22674 server_base.cc:1034] running on GCE node
W20260925 10:08:39.395781 22679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:39.396446 22674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:39.396737 22674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:39.397935 22674 hybrid_clock.cc:648] HybridClock initialized: now 1790330919397910 us; error 42 us; skew 500 ppm
Sep 25 10:08:39 dist-test-slave-s6tm krb5kdc[22661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330919, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:39.404874 22674 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:08:39.405458 22674 webserver.cc:460] Webserver started at http://127.14.41.126:41595/ using document root <none> and password file <none>
I20260925 10:08:39.405725 22674 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:39.405786 22674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:39.405946 22674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:39.407068 22674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "123099101e2d42f6a3ebf7350d9cffa5"
format_stamp: "Formatted at 2026-09-25 10:08:39 on dist-test-slave-s6tm"
server_key: "8447d9d97403700fa79cf25118cf09b3"
server_key_iv: "4c0196765b1a3f5fe1f860c7dbca8a12"
server_key_version: "encryptionkey@0"
I20260925 10:08:39.407495 22674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "123099101e2d42f6a3ebf7350d9cffa5"
format_stamp: "Formatted at 2026-09-25 10:08:39 on dist-test-slave-s6tm"
server_key: "8447d9d97403700fa79cf25118cf09b3"
server_key_iv: "4c0196765b1a3f5fe1f860c7dbca8a12"
server_key_version: "encryptionkey@0"
I20260925 10:08:39.408951 22674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:08:39.409904 22689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:39.410120 22674 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:39.410226 22674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "123099101e2d42f6a3ebf7350d9cffa5"
format_stamp: "Formatted at 2026-09-25 10:08:39 on dist-test-slave-s6tm"
server_key: "8447d9d97403700fa79cf25118cf09b3"
server_key_iv: "4c0196765b1a3f5fe1f860c7dbca8a12"
server_key_version: "encryptionkey@0"
I20260925 10:08:39.410347 22674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:39.421960 22674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:39.424329 22674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:39.424499 22674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:39.428928 22674 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:46043
I20260925 10:08:39.428956 22741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:46043 every 8 connection(s)
I20260925 10:08:39.429332 22674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:39.430207 22742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:39.432718 22742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5: Bootstrap starting.
I20260925 10:08:39.433357 22742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:39.433648 22742 log.cc:824] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:39.434422 22742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5: No bootstrap required, opened a new log
I20260925 10:08:39.436192 22742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123099101e2d42f6a3ebf7350d9cffa5" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46043 } }
I20260925 10:08:39.436358 22742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:39.436403 22742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123099101e2d42f6a3ebf7350d9cffa5, State: Initialized, Role: FOLLOWER
I20260925 10:08:39.436566 22742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123099101e2d42f6a3ebf7350d9cffa5" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46043 } }
I20260925 10:08:39.436650 22742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:08:39.436693 22742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:39.436746 22742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:39.437338 22742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123099101e2d42f6a3ebf7350d9cffa5" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46043 } }
I20260925 10:08:39.437460 22742 leader_election.cc:302] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 123099101e2d42f6a3ebf7350d9cffa5; no voters:
I20260925 10:08:39.437649 22742 leader_election.cc:288] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:08:39.437697 22745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:39.437924 22742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:39.437970 22745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [term 1 LEADER]: Becoming Leader. State: Replica: 123099101e2d42f6a3ebf7350d9cffa5, State: Running, Role: LEADER
I20260925 10:08:39.438172 22745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123099101e2d42f6a3ebf7350d9cffa5" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46043 } }
I20260925 10:08:39.438808 22747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 123099101e2d42f6a3ebf7350d9cffa5. Latest consensus state: current_term: 1 leader_uuid: "123099101e2d42f6a3ebf7350d9cffa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123099101e2d42f6a3ebf7350d9cffa5" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46043 } } }
I20260925 10:08:39.438835 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 22674
I20260925 10:08:39.438894 22747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:39.438961 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:08:39.439169 14501 external_mini_cluster.cc:1428] Setting key ae6df3f35e295a258db6d87b32e52399
I20260925 10:08:39.439272 22746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "123099101e2d42f6a3ebf7350d9cffa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123099101e2d42f6a3ebf7350d9cffa5" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46043 } } }
I20260925 10:08:39.439358 22746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [sys.catalog]: This master's current role is: LEADER
W20260925 10:08:39.439971 22760 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:08:39.440076 22760 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:08:39.440215 22754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:39.440385 22754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:39.445766 22754 catalog_manager.cc:1347] Generated new cluster ID: 03b8baaa18c741529e30b811701fa687
I20260925 10:08:39.445829 22754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:08:39 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:39.455698 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:39.439706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59148 (local address 127.14.41.126:46043)
0925 10:08:39.444086 (+ 4380us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:39.444097 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:39.444143 (+ 46us) server_negotiation.cc:407] Connection header received
0925 10:08:39.444279 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:39.444287 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:39.444347 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:39.444530 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:39.445231 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:39.447055 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:39.448148 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:39.448841 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:39.448960 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:39.451824 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:39.451829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:39.451829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:39.451844 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:39.453944 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:39.454376 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:39.454378 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:39.454379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:39.454499 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:39.454708 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:39.454708 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:39.454709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:39.455026 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:39.455167 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:39.455300 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 10:08:39.455454 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4314,"thread_start_us":68,"threads_started":1}
I20260925 10:08:39.456873 22754 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:08:39.457427 22754 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:39.464052 22754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5: Generated new TSK 0
I20260925 10:08:39.464298 22754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
I20260925 10:08:39.498437 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46043
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:44837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:39.591310 22768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:39.591547 22768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:39.591609 22768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:39.593084 22768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:39.593137 22768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:39.593189 22768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:08:39.594913 22768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:44837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:39.595203 22768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:39.595465 22768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:39.601012 22776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:39.601013 22774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:39.601012 22773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:39.601320 22768 server_base.cc:1034] running on GCE node
I20260925 10:08:39.601542 22768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:39.601764 22768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:39.602960 22768 hybrid_clock.cc:648] HybridClock initialized: now 1790330919602944 us; error 31 us; skew 500 ppm
Sep 25 10:08:39 dist-test-slave-s6tm krb5kdc[22661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330919, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:39.605723 22768 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:08:39.606261 22768 webserver.cc:460] Webserver started at http://127.14.41.65:42387/ using document root <none> and password file <none>
I20260925 10:08:39.606521 22768 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:39.606591 22768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:39.606762 22768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:39.607899 22768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "d68c145661214c8d8ed1fd5bb090630b"
format_stamp: "Formatted at 2026-09-25 10:08:39 on dist-test-slave-s6tm"
server_key: "690da12f199c50e3a749ecf0fd0680f4"
server_key_iv: "c3a3bef7784a4f4b2378bda90cf548ff"
server_key_version: "encryptionkey@0"
I20260925 10:08:39.608337 22768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "d68c145661214c8d8ed1fd5bb090630b"
format_stamp: "Formatted at 2026-09-25 10:08:39 on dist-test-slave-s6tm"
server_key: "690da12f199c50e3a749ecf0fd0680f4"
server_key_iv: "c3a3bef7784a4f4b2378bda90cf548ff"
server_key_version: "encryptionkey@0"
I20260925 10:08:39.609866 22768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:39.610658 22783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:39.610814 22768 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:08:39.610908 22768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "d68c145661214c8d8ed1fd5bb090630b"
format_stamp: "Formatted at 2026-09-25 10:08:39 on dist-test-slave-s6tm"
server_key: "690da12f199c50e3a749ecf0fd0680f4"
server_key_iv: "c3a3bef7784a4f4b2378bda90cf548ff"
server_key_version: "encryptionkey@0"
I20260925 10:08:39.610993 22768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:39.644799 22768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:39.647583 22768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:39.647758 22768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:39.648061 22768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:39.648408 22768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:39.648463 22768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:39.648504 22768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:39.648530 22768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:39.655114 22768 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:43217
I20260925 10:08:39.655164 22896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:43217 every 8 connection(s)
I20260925 10:08:39.655511 22768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
Sep 25 10:08:39 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330919, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:39.664911 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 22768
I20260925 10:08:39.665038 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:08:39.665266 14501 external_mini_cluster.cc:1428] Setting key 43278b0533b67ac98d63c6dad72caade
I20260925 10:08:39.665647 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:39.656280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:56143 (local address 127.14.41.126:46043)
0925 10:08:39.656355 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:39.656358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:39.656588 (+ 230us) server_negotiation.cc:407] Connection header received
0925 10:08:39.657101 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:39.657102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:39.657114 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:39.657178 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:39.658160 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:39.658701 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:39.659785 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:39.660082 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:39.660145 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:39.662458 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:39.662459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:39.662460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:39.662467 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:39.664134 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:39.664527 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:39.664528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:39.664528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:39.664583 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:39.664918 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:39.664919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:39.664919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:39.665077 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:39.665153 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:39.665452 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 10:08:39.665557 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 10:08:39.665843 22899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:39.656255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.65:56143)
0925 10:08:39.656440 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:39.656485 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:39.656991 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:39.657237 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:39.657241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:39.657842 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:39.658040 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:39.658045 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:39.658828 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:39.658829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:39.659673 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:39.659675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:39.660156 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:39.660157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:39.660284 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:39.660898 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:39.660907 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:39.662347 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:39.664211 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:39.664217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:39.664219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:39.664452 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:39.664653 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:39.664653 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:39.664654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:39.664843 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:39.665143 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:39.665149 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:39.665279 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 10:08:39.665408 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 10:08:39.665575 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":105,"threads_started":1}
I20260925 10:08:39.666380 22897 heartbeater.cc:347] Connected to a master server at 127.14.41.126:46043
I20260925 10:08:39.666531 22897 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:39.666720 22897 heartbeater.cc:511] Master 127.14.41.126:46043 requested a full tablet report, sending...
I20260925 10:08:39.667428 22706 ts_manager.cc:194] Registered new tserver with Master: d68c145661214c8d8ed1fd5bb090630b (127.14.41.65:43217)
I20260925 10:08:39.668154 22706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:56143
WARNING: no policy specified for kudu/127.14.41.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
I20260925 10:08:39.695642 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46043
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:44837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:39.783986 22902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:39.784253 22902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:39.784296 22902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:39.785586 22902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:39.785679 22902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:39.785725 22902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:08:39.787216 22902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:44837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:39.787501 22902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:39.787739 22902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:39.792996 22907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:39.793017 22910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:39.793027 22908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:39.793305 22902 server_base.cc:1034] running on GCE node
I20260925 10:08:39.793556 22902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:39.793817 22902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:39.795007 22902 hybrid_clock.cc:648] HybridClock initialized: now 1790330919794992 us; error 29 us; skew 500 ppm
Sep 25 10:08:39 dist-test-slave-s6tm krb5kdc[22661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330919, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:39.797655 22902 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:08:39.798199 22902 webserver.cc:460] Webserver started at http://127.14.41.66:45559/ using document root <none> and password file <none>
I20260925 10:08:39.798444 22902 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:39.798514 22902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:39.798672 22902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:39.799790 22902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "ee6dd6d0a8fa4ac0b5d1ecdf6c8b2d8e"
format_stamp: "Formatted at 2026-09-25 10:08:39 on dist-test-slave-s6tm"
server_key: "f826f567696856c4e0ff1d05123a6e39"
server_key_iv: "e317442504dade7d6545ae4d1c0da4af"
server_key_version: "encryptionkey@0"
I20260925 10:08:39.800197 22902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "ee6dd6d0a8fa4ac0b5d1ecdf6c8b2d8e"
format_stamp: "Formatted at 2026-09-25 10:08:39 on dist-test-slave-s6tm"
server_key: "f826f567696856c4e0ff1d05123a6e39"
server_key_iv: "e317442504dade7d6545ae4d1c0da4af"
server_key_version: "encryptionkey@0"
I20260925 10:08:39.801673 22902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:39.802547 22917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:39.802718 22902 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:08:39.802836 22902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "ee6dd6d0a8fa4ac0b5d1ecdf6c8b2d8e"
format_stamp: "Formatted at 2026-09-25 10:08:39 on dist-test-slave-s6tm"
server_key: "f826f567696856c4e0ff1d05123a6e39"
server_key_iv: "e317442504dade7d6545ae4d1c0da4af"
server_key_version: "encryptionkey@0"
I20260925 10:08:39.802917 22902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:39.824074 22902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:39.826624 22902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:39.826825 22902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:39.827088 22902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:39.827427 22902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:39.827484 22902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:39.827526 22902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:39.827553 22902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:39.833904 22902 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:46589
I20260925 10:08:39.834010 23030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:46589 every 8 connection(s)
I20260925 10:08:39.834369 22902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:08:39 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330919, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:39.841363 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 22902
I20260925 10:08:39.841485 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:08:39.841686 14501 external_mini_cluster.cc:1428] Setting key d20cdf4d43427ceecad5372f38104413
I20260925 10:08:39.845295 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:39.835164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:35167 (local address 127.14.41.126:46043)
0925 10:08:39.835224 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:39.835226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:39.835453 (+ 227us) server_negotiation.cc:407] Connection header received
0925 10:08:39.836007 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:39.836009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:39.836021 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:39.836094 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:39.837050 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:39.837634 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:39.838928 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:39.839195 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:39.839259 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:39.841615 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:39.841617 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:39.841617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:39.841624 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:39.843587 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:39.844251 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:39.844252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:39.844253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:39.844320 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:39.844610 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:39.844612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:39.844612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:39.844743 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:39.844822 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:39.845114 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 10:08:39.845202 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 10:08:39.845504 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:39.835163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.66:35167)
0925 10:08:39.835320 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:39.835358 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:39.835857 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:39.836150 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:39.836154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:39.836759 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:39.836958 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:39.836963 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:39.837723 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:39.837725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:39.838851 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:39.838854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:39.839282 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:39.839283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:39.839387 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:39.839957 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:39.839966 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:39.841494 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:39.843693 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:39.843701 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:39.843703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:39.844148 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:39.844416 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:39.844416 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:39.844417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:39.844536 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:39.844812 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:39.844819 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:39.844972 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 10:08:39.845090 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 10:08:39.845260 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260925 10:08:39.846108 23031 heartbeater.cc:347] Connected to a master server at 127.14.41.126:46043
I20260925 10:08:39.846235 23031 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:39.846488 23031 heartbeater.cc:511] Master 127.14.41.126:46043 requested a full tablet report, sending...
I20260925 10:08:39.846972 22706 ts_manager.cc:194] Registered new tserver with Master: ee6dd6d0a8fa4ac0b5d1ecdf6c8b2d8e (127.14.41.66:46589)
I20260925 10:08:39.847393 22706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:35167
WARNING: no policy specified for kudu/127.14.41.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
I20260925 10:08:39.871381 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46043
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:44837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:39.954754 23036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:39.954993 23036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:39.955017 23036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:39.956254 23036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:39.956300 23036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:39.956328 23036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:08:39.957752 23036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:44837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:39.957969 23036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:39.958156 23036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:39.963716 23041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:39.963721 23042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:39.964044 23036 server_base.cc:1034] running on GCE node
W20260925 10:08:39.964149 23044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:39.964401 23036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:39.964720 23036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:39.965924 23036 hybrid_clock.cc:648] HybridClock initialized: now 1790330919965905 us; error 32 us; skew 500 ppm
Sep 25 10:08:39 dist-test-slave-s6tm krb5kdc[22661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330919, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:39.969240 23036 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:08:39.969797 23036 webserver.cc:460] Webserver started at http://127.14.41.67:33757/ using document root <none> and password file <none>
I20260925 10:08:39.970075 23036 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:39.970127 23036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:39.970229 23036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:39.971387 23036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "c93012f41d8242dfb9e8b46db1bd8503"
format_stamp: "Formatted at 2026-09-25 10:08:39 on dist-test-slave-s6tm"
server_key: "98cbb9af6580497386a80dc2c456ba20"
server_key_iv: "29c790fb0b81cbed020c32f295691923"
server_key_version: "encryptionkey@0"
I20260925 10:08:39.971778 23036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "c93012f41d8242dfb9e8b46db1bd8503"
format_stamp: "Formatted at 2026-09-25 10:08:39 on dist-test-slave-s6tm"
server_key: "98cbb9af6580497386a80dc2c456ba20"
server_key_iv: "29c790fb0b81cbed020c32f295691923"
server_key_version: "encryptionkey@0"
I20260925 10:08:39.973492 23036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:39.974555 23051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:39.974932 23036 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:39.975023 23036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "c93012f41d8242dfb9e8b46db1bd8503"
format_stamp: "Formatted at 2026-09-25 10:08:39 on dist-test-slave-s6tm"
server_key: "98cbb9af6580497386a80dc2c456ba20"
server_key_iv: "29c790fb0b81cbed020c32f295691923"
server_key_version: "encryptionkey@0"
I20260925 10:08:39.975132 23036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:39.990502 23036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:39.992841 23036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:39.993090 23036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:39.993337 23036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:39.993667 23036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:39.993702 23036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:39.993724 23036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:39.993772 23036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:39.999847 23036 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:40873
I20260925 10:08:39.999888 23164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:40873 every 8 connection(s)
I20260925 10:08:40.000283 23036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:08:40.006580 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 23036
I20260925 10:08:40.006692 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:08:40.006868 14501 external_mini_cluster.cc:1428] Setting key b2e193854faa6359ac8227e8ee7c900a
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330919, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:40.010725 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:40.001106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:47729 (local address 127.14.41.126:46043)
0925 10:08:40.001194 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:40.001197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:40.001430 (+ 233us) server_negotiation.cc:407] Connection header received
0925 10:08:40.002065 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:40.002066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:40.002077 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:40.002144 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:40.003273 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.003804 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.004947 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.005143 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.005200 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.008007 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:40.008008 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:40.008008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:40.008015 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:40.009365 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:40.009771 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:40.009772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:40.009772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:40.009831 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:40.010058 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:40.010059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:40.010059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:40.010170 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:40.010221 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:40.010487 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 10:08:40.010564 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:08:40.011016 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:40.001067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.67:47729)
0925 10:08:40.001301 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:40.001342 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:40.001959 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:40.002259 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:40.002263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:40.002968 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:40.003169 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:40.003175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.003918 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.003919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:40.004863 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:40.004865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.005225 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.005226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:40.005350 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.005966 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:40.005979 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:40.007879 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:40.009444 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:40.009451 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:40.009453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:40.009679 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:40.009898 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:40.009899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:40.009900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:40.010000 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:40.010223 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:40.010227 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:40.010345 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 10:08:40.010434 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 10:08:40.010591 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":112,"threads_started":1}
I20260925 10:08:40.011468 23165 heartbeater.cc:347] Connected to a master server at 127.14.41.126:46043
I20260925 10:08:40.011581 23165 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:40.011765 23165 heartbeater.cc:511] Master 127.14.41.126:46043 requested a full tablet report, sending...
I20260925 10:08:40.012207 22706 ts_manager.cc:194] Registered new tserver with Master: c93012f41d8242dfb9e8b46db1bd8503 (127.14.41.67:40873)
I20260925 10:08:40.012622 22706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:47729
I20260925 10:08:40.019042 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.65.keytab.
Entry for principal oryx/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.65.keytab.
WARNING: no policy specified for oryx/127.14.41.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.41.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.66.keytab.
Entry for principal oryx/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.66.keytab.
WARNING: no policy specified for oryx/127.14.41.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.14.41.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.67.keytab.
Entry for principal oryx/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.67.keytab.
I20260925 10:08:40.098817 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 22768
I20260925 10:08:40.103586 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 22902
I20260925 10:08:40.108134 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 23036
I20260925 10:08:40.112409 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 22674
I20260925 10:08:40.117700 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:46043
--webserver_interface=127.14.41.126
--webserver_port=41595
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:44837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:46043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:40.202158 23174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:40.202396 23174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:40.202421 23174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:40.203583 23174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:40.203626 23174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:40.203640 23174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:40.203653 23174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:40.203665 23174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:08:40.205137 23174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:44837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:46043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:46043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=41595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:40.205349 23174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:40.205540 23174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:40.210937 23179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:40.211105 23174 server_base.cc:1034] running on GCE node
W20260925 10:08:40.210942 23182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:40.211264 23180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:40.211632 23174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:40.212044 23174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:40.213224 23174 hybrid_clock.cc:648] HybridClock initialized: now 1790330920213199 us; error 45 us; skew 500 ppm
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330920, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:40.216156 23174 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:08:40.216677 23174 webserver.cc:460] Webserver started at http://127.14.41.126:41595/ using document root <none> and password file <none>
I20260925 10:08:40.217063 23174 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:40.217145 23174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:40.219079 23174 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:08:40.220018 23189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:40.220184 23174 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:40.220286 23174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "123099101e2d42f6a3ebf7350d9cffa5"
format_stamp: "Formatted at 2026-09-25 10:08:39 on dist-test-slave-s6tm"
server_key: "8447d9d97403700fa79cf25118cf09b3"
server_key_iv: "4c0196765b1a3f5fe1f860c7dbca8a12"
server_key_version: "encryptionkey@0"
I20260925 10:08:40.220656 23174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:40.243647 23174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:40.246104 23174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:40.246230 23174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:40.250380 23174 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:46043
I20260925 10:08:40.250464 23241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:46043 every 8 connection(s)
I20260925 10:08:40.250878 23174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:40.252497 23242 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:08:40.253202 23242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5: Bootstrap starting.
I20260925 10:08:40.253201 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 23174
I20260925 10:08:40.253571 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:43217
--local_ip_for_outbound_sockets=127.14.41.65
--tserver_master_addrs=127.14.41.126:46043
--webserver_port=42387
--webserver_interface=127.14.41.65
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:44837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.41.65
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:08:40.262179 23242 log.cc:824] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:40.264170 23242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:08:40.264492 23242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5: Bootstrap complete.
I20260925 10:08:40.266789 23242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123099101e2d42f6a3ebf7350d9cffa5" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46043 } }
I20260925 10:08:40.267066 23242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123099101e2d42f6a3ebf7350d9cffa5, State: Initialized, Role: FOLLOWER
I20260925 10:08:40.267285 23242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123099101e2d42f6a3ebf7350d9cffa5" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46043 } }
I20260925 10:08:40.267397 23242 raft_consensus.cc:388] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:08:40.267453 23242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:40.267503 23242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:08:40.268666 23242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123099101e2d42f6a3ebf7350d9cffa5" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46043 } }
I20260925 10:08:40.268802 23242 leader_election.cc:302] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 123099101e2d42f6a3ebf7350d9cffa5; no voters:
I20260925 10:08:40.269030 23242 leader_election.cc:288] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:08:40.269070 23246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:08:40.269184 23246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [term 2 LEADER]: Becoming Leader. State: Replica: 123099101e2d42f6a3ebf7350d9cffa5, State: Running, Role: LEADER
I20260925 10:08:40.269313 23242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:40.269521 23246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123099101e2d42f6a3ebf7350d9cffa5" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46043 } }
I20260925 10:08:40.269912 23247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "123099101e2d42f6a3ebf7350d9cffa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123099101e2d42f6a3ebf7350d9cffa5" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46043 } } }
I20260925 10:08:40.269936 23248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 123099101e2d42f6a3ebf7350d9cffa5. Latest consensus state: current_term: 2 leader_uuid: "123099101e2d42f6a3ebf7350d9cffa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123099101e2d42f6a3ebf7350d9cffa5" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46043 } } }
I20260925 10:08:40.270117 23247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:40.270134 23248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:40.270491 23256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:40.271104 23256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:40.271302 23256 catalog_manager.cc:1259] Loaded cluster ID: 03b8baaa18c741529e30b811701fa687
I20260925 10:08:40.271381 23256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:08:40.272648 23256 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:40.272922 23256 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 123099101e2d42f6a3ebf7350d9cffa5: Loaded TSK: 0
I20260925 10:08:40.273202 23256 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:08:40.341693 23244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:40.341928 23244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:40.341953 23244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:40.343475 23244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:40.343537 23244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:40.343567 23244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:08:40.345443 23244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:44837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.65.keytab
--principal=oryx/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:43217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=42387
--tserver_master_addrs=127.14.41.126:46043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:40.345837 23244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:40.346113 23244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:08:40.351632 23244 server_base.cc:1034] running on GCE node
W20260925 10:08:40.351739 23269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:40.351612 23271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:40.351601 23268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:40.352145 23244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:40.352478 23244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:40.353657 23244 hybrid_clock.cc:648] HybridClock initialized: now 1790330920353623 us; error 50 us; skew 500 ppm
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330920, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:40.356664 23244 init.cc:377] Logged in from keytab as oryx/127.14.41.65@KRBTEST.COM (short username oryx)
I20260925 10:08:40.357221 23244 webserver.cc:460] Webserver started at http://127.14.41.65:42387/ using document root <none> and password file <none>
I20260925 10:08:40.357527 23244 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:40.357630 23244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:40.359366 23244 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:40.360177 23278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:40.360391 23244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:40.360487 23244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "d68c145661214c8d8ed1fd5bb090630b"
format_stamp: "Formatted at 2026-09-25 10:08:39 on dist-test-slave-s6tm"
server_key: "690da12f199c50e3a749ecf0fd0680f4"
server_key_iv: "c3a3bef7784a4f4b2378bda90cf548ff"
server_key_version: "encryptionkey@0"
I20260925 10:08:40.360875 23244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:40.375743 23244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:40.378651 23244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:40.378940 23244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:40.379299 23244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:40.379801 23244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:40.379842 23244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:40.379866 23244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:40.379879 23244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:40.387666 23244 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:43217
I20260925 10:08:40.387748 23391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:43217 every 8 connection(s)
I20260925 10:08:40.388206 23244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:08:40.389490 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 23244
I20260925 10:08:40.389843 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:46589
--local_ip_for_outbound_sockets=127.14.41.66
--tserver_master_addrs=127.14.41.126:46043
--webserver_port=45559
--webserver_interface=127.14.41.66
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:44837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.41.66
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:40.398463 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:40.389023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.65:57381)
0925 10:08:40.389252 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:40.389303 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:40.389980 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:40.391557 (+ 1577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:40.391561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:40.392261 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:40.392467 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:40.392482 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.393677 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.393679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:40.394502 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:40.394505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.395138 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.395139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:40.395280 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.395863 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:40.395873 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:40.398219 (+ 2346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":120,"threads_started":1}
W20260925 10:08:40.398739 23392 heartbeater.cc:644] Failed to heartbeat to 127.14.41.126:46043 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.41.126:46043: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
W20260925 10:08:40.399034 23395 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:40.389170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:57381 (local address 127.14.41.126:46043)
0925 10:08:40.391103 (+ 1933us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:40.391112 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:40.391147 (+ 35us) server_negotiation.cc:407] Connection header received
0925 10:08:40.391272 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:40.391278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:40.391324 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:40.391494 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:40.392622 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.393552 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.394616 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.395017 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.395143 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.398360 (+ 3217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.65:57381: BlockingRecv error: recv got EOF from 127.14.41.65:57381 (error 108)
Metrics: {"server-negotiator.queue_time_us":1877,"thread_start_us":94,"threads_started":1}
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:40.403841 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:40.398952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.65:52079)
0925 10:08:40.399304 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:40.399318 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:40.399369 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:40.399547 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:40.399548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:40.399674 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:40.399734 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:40.399736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.400460 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.400461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:40.400976 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:40.400978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.401361 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.401361 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:40.401433 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.401928 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:40.401933 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:40.403746 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":87,"threads_started":1}
W20260925 10:08:40.403889 23395 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:40.399142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:52079 (local address 127.14.41.126:46043)
0925 10:08:40.399347 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:40.399349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:40.399358 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:08:40.399442 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:40.399443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:40.399458 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:40.399794 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:40.399823 (+ 29us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.400370 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.401083 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.401276 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.401363 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.403822 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.65:52079: BlockingRecv error: recv got EOF from 127.14.41.65:52079 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":96,"threads_started":1}
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:40.409067 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:40.404406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.65:50835)
0925 10:08:40.404450 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:40.404461 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:40.404497 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:40.404676 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:40.404677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:40.404811 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:40.404840 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:40.404841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.405476 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.405477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:40.405978 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:40.405979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.406362 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.406363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:40.406422 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.406948 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:40.406954 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:40.408970 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 10:08:40.409126 23395 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:40.404452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:50835 (local address 127.14.41.126:46043)
0925 10:08:40.404563 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:40.404564 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:40.404570 (+ 6us) server_negotiation.cc:407] Connection header received
0925 10:08:40.404588 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:40.404588 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:40.404597 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:40.404658 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:40.404925 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.405394 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.406083 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.406285 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.406363 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.409054 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.65:50835: BlockingRecv error: recv got EOF from 127.14.41.65:50835 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260925 10:08:40.409216 23392 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:08:40.478626 23396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:40.478910 23396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:40.478955 23396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:40.480264 23396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:40.480329 23396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:40.480373 23396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:08:40.481850 23396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:44837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.66.keytab
--principal=oryx/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:46589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=45559
--tserver_master_addrs=127.14.41.126:46043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:40.482153 23396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:40.482393 23396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:40.487880 23403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:40.487892 23406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:40.487882 23404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:40.488008 23396 server_base.cc:1034] running on GCE node
I20260925 10:08:40.488412 23396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:40.488669 23396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:40.489861 23396 hybrid_clock.cc:648] HybridClock initialized: now 1790330920489848 us; error 33 us; skew 500 ppm
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330920, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:40.492702 23396 init.cc:377] Logged in from keytab as oryx/127.14.41.66@KRBTEST.COM (short username oryx)
I20260925 10:08:40.493256 23396 webserver.cc:460] Webserver started at http://127.14.41.66:45559/ using document root <none> and password file <none>
I20260925 10:08:40.493495 23396 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:40.493579 23396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:40.495015 23396 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 10:08:40.495622 23413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:40.495813 23396 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:40.495996 23396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "ee6dd6d0a8fa4ac0b5d1ecdf6c8b2d8e"
format_stamp: "Formatted at 2026-09-25 10:08:39 on dist-test-slave-s6tm"
server_key: "f826f567696856c4e0ff1d05123a6e39"
server_key_iv: "e317442504dade7d6545ae4d1c0da4af"
server_key_version: "encryptionkey@0"
I20260925 10:08:40.496366 23396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:40.514010 23396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:40.516561 23396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:40.516739 23396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:40.517006 23396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:40.517329 23396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:40.517385 23396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:40.517432 23396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:40.517458 23396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:40.524165 23396 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:46589
I20260925 10:08:40.524209 23526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:46589 every 8 connection(s)
I20260925 10:08:40.524632 23396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:08:40.525992 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 23396
I20260925 10:08:40.526320 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:40873
--local_ip_for_outbound_sockets=127.14.41.67
--tserver_master_addrs=127.14.41.126:46043
--webserver_port=33757
--webserver_interface=127.14.41.67
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:44837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.14.41.67
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:08:40.534636 23395 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:40.525444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:50265 (local address 127.14.41.126:46043)
0925 10:08:40.525518 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:40.525520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:40.525750 (+ 230us) server_negotiation.cc:407] Connection header received
0925 10:08:40.526575 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:40.526581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:40.526595 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:40.526641 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:40.528060 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.528880 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.530634 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.530851 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.530915 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.534547 (+ 3632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.66:50265: BlockingRecv error: recv got EOF from 127.14.41.66:50265 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:08:40.534709 23529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:40.525446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.66:50265)
0925 10:08:40.525606 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:40.525644 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:40.526455 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:40.526724 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:40.526729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:40.527608 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:40.527962 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:40.527970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.529032 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.529033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:40.530518 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:40.530522 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.530958 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.530959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:40.531141 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.531780 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:40.531792 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:40.534470 (+ 2678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
W20260925 10:08:40.534948 23527 heartbeater.cc:644] Failed to heartbeat to 127.14.41.126:46043 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.41.126:46043: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:40.539757 23529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:40.535217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.66:56201)
0925 10:08:40.535272 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:40.535283 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:40.535327 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:40.535479 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:40.535480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:40.535621 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:40.535670 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:40.535672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.536408 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.536410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:40.536923 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:40.536925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.537309 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.537310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:40.537384 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.537866 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:40.537872 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:40.539676 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 10:08:40.540016 23395 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:40.535291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:56201 (local address 127.14.41.126:46043)
0925 10:08:40.535344 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:40.535346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:40.535357 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:08:40.535386 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:40.535387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:40.535397 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:40.535471 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:40.535754 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.536320 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.537014 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.537221 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.537310 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.539782 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.66:56201: BlockingRecv error: recv got EOF from 127.14.41.66:56201 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:40.545265 23529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:40.540269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.66:43045)
0925 10:08:40.540324 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:40.540337 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:40.540395 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:40.540549 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:40.540550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:40.540705 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:40.540746 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:40.540747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.541385 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.541386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:40.542217 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:40.542219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.542586 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.542587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:40.542645 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.543109 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:40.543115 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:40.545170 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 10:08:40.545320 23395 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:40.540314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:43045 (local address 127.14.41.126:46043)
0925 10:08:40.540373 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:40.540375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:40.540384 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:08:40.540473 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:40.540474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:40.540483 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:40.540531 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:40.540812 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.541301 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.542315 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.542507 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.542566 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.545256 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.66:43045: BlockingRecv error: recv got EOF from 127.14.41.66:43045 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 10:08:40.545426 23527 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:08:40.614610 23530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:40.614902 23530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:40.614948 23530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:40.616304 23530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:40.616380 23530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:40.616430 23530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:08:40.618130 23530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:44837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/krb5kdc/oryx_127.14.41.67.keytab
--principal=oryx/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:40873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=33757
--tserver_master_addrs=127.14.41.126:46043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:40.618453 23530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:40.618733 23530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:08:40.624360 23530 server_base.cc:1034] running on GCE node
W20260925 10:08:40.624326 23536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:40.624320 23535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:40.624320 23538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:40.624881 23530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:40.625211 23530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:40.626408 23530 hybrid_clock.cc:648] HybridClock initialized: now 1790330920626378 us; error 41 us; skew 500 ppm
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330920, etypes {rep=17 tkt=17 ses=17}, oryx/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:40.629511 23530 init.cc:377] Logged in from keytab as oryx/127.14.41.67@KRBTEST.COM (short username oryx)
I20260925 10:08:40.629944 23530 webserver.cc:460] Webserver started at http://127.14.41.67:33757/ using document root <none> and password file <none>
I20260925 10:08:40.630162 23530 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:40.630220 23530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:40.631640 23530 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:40.632416 23545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:40.632591 23530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:08:40.632697 23530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "c93012f41d8242dfb9e8b46db1bd8503"
format_stamp: "Formatted at 2026-09-25 10:08:39 on dist-test-slave-s6tm"
server_key: "98cbb9af6580497386a80dc2c456ba20"
server_key_iv: "29c790fb0b81cbed020c32f295691923"
server_key_version: "encryptionkey@0"
I20260925 10:08:40.633077 23530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:40.654908 23530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:40.657171 23530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:40.657290 23530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:40.657490 23530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:40.657828 23530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:40.657867 23530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:40.657891 23530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:40.657903 23530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:40.664139 23530 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:40873
I20260925 10:08:40.664191 23658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:40873 every 8 connection(s)
I20260925 10:08:40.664597 23530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:40.673085 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 23530
W20260925 10:08:40.673108 23395 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:40.665590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:50855 (local address 127.14.41.126:46043)
0925 10:08:40.665678 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:40.665680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:40.665895 (+ 215us) server_negotiation.cc:407] Connection header received
0925 10:08:40.666436 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:40.666437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:40.666447 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:40.666512 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:40.667504 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.668175 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.669337 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.669652 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.669783 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.673050 (+ 3267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.67:50855: BlockingRecv error: recv got EOF from 127.14.41.67:50855 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:08:40.673174 23661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:40.665551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.67:50855)
0925 10:08:40.665749 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:40.665801 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:40.666342 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:40.666598 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:40.666601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:40.667181 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:40.667404 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:40.667409 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.668267 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.668268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:40.669232 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:40.669235 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.669801 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.669802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:40.669960 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.670599 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:40.670609 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:40.672969 (+ 2360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":115,"threads_started":1}
W20260925 10:08:40.673478 23659 heartbeater.cc:644] Failed to heartbeat to 127.14.41.126:46043 (0 consecutive failures): Not authorized: Failed to ping master at 127.14.41.126:46043: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:40.679026 23661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:40.673693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.67:33837)
0925 10:08:40.673747 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:40.673758 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:40.673798 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:40.673970 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:40.673971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:40.674117 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:40.674184 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:40.674186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.675039 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.675039 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:40.675859 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:40.675862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.676305 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.676306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:40.676372 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.676828 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:40.676833 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:40.678939 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 10:08:40.679092 23395 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:40.673724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:33837 (local address 127.14.41.126:46043)
0925 10:08:40.673795 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:40.673796 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:40.673806 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:08:40.673857 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:40.673858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:40.673866 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:40.673956 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:40.674285 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.674957 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.676012 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.676206 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.676291 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.678998 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.67:33837: BlockingRecv error: recv got EOF from 127.14.41.67:33837 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:08:40.683209 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:40.673694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59152 (local address 127.14.41.126:46043)
0925 10:08:40.673784 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:40.673787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:40.673897 (+ 110us) server_negotiation.cc:407] Connection header received
0925 10:08:40.673969 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:40.673970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:40.673982 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:40.674053 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:40.674709 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.675274 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.676301 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.676498 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.676577 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.678037 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:40.678042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:40.678042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:40.678055 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:40.681399 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:40.681938 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:40.681942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:40.681942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:40.682089 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:40.682332 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:40.682333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:40.682334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:40.682700 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:40.682849 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:40.682962 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 10:08:40.683090 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:40 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:40.684578 23661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:40.679540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.67:32785)
0925 10:08:40.679592 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:40.679604 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:40.679641 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:40.679788 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:40.679789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:40.679988 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:40.680031 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:40.680033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.680850 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.680851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:40.681603 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:40.681606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.682058 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.682059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:40.682128 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.682503 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:40.682509 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:40.684528 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 10:08:40.684716 23659 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:08:40.684769 23395 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:40.679601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:32785 (local address 127.14.41.126:46043)
0925 10:08:40.679640 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:40.679642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:40.679651 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:08:40.679701 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:40.679702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:40.679711 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:40.679783 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:40.680104 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.680760 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.681683 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:40.681977 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:40.682036 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:40.684705 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.67:32785: BlockingRecv error: recv got EOF from 127.14.41.67:32785 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 25 10:08:41 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:41 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:41.415117 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:41.409662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.65:45201)
0925 10:08:41.409826 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:41.409840 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:41.409920 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:41.410258 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:41.410259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:41.410419 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:41.410471 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:41.410473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:41.411168 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:41.411169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:41.411800 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:41.411802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:41.412209 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:41.412211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:41.412264 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:41.412748 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:41.412755 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:41.415041 (+ 2286us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":59,"threads_started":1}
W20260925 10:08:41.415257 23664 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:41.409952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:45201 (local address 127.14.41.126:46043)
0925 10:08:41.410086 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:41.410088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:41.410096 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:08:41.410121 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:41.410121 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:41.410131 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:41.410193 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:41.410559 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:41.411081 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:41.411895 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:41.412120 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:41.412177 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:41.415144 (+ 2967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.65:45201: BlockingRecv error: recv got EOF from 127.14.41.65:45201 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
Sep 25 10:08:41 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:41 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:41.551007 23665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:41.545900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.66:44403)
0925 10:08:41.546046 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:41.546058 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:41.546126 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:41.546284 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:41.546285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:41.546490 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:41.546540 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:41.546543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:41.547265 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:41.547266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:41.547838 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:41.547840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:41.548292 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:41.548293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:41.548349 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:41.549019 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:41.549025 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:41.550895 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
W20260925 10:08:41.551043 23664 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:41.546044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:44403 (local address 127.14.41.126:46043)
0925 10:08:41.546121 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:41.546123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:41.546135 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:08:41.546187 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:41.546188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:41.546202 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:41.546277 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:41.546650 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:41.547170 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:41.547964 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:41.548220 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:41.548293 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:41.550976 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.66:44403: BlockingRecv error: recv got EOF from 127.14.41.66:44403 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 10:08:41 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:41 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:41.690294 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:41.685249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.67:36385)
0925 10:08:41.685398 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:41.685410 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:41.685478 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:41.685648 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:41.685649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:41.685804 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:41.685856 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:41.685858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:41.686603 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:41.686604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:41.687205 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:41.687207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:41.687613 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:41.687614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:41.687659 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:41.688214 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:41.688218 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:41.690224 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
W20260925 10:08:41.690407 23664 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:41.685269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:36385 (local address 127.14.41.126:46043)
0925 10:08:41.685384 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:41.685386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:41.685461 (+ 75us) server_negotiation.cc:407] Connection header received
0925 10:08:41.685536 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:41.685536 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:41.685554 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:41.685602 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:41.685967 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:41.686525 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:41.687317 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:41.687533 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:41.687598 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:41.690327 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.67:36385: BlockingRecv error: recv got EOF from 127.14.41.67:36385 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 25 10:08:42 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:42 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:42.421054 23667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:42.415704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.65:54177)
0925 10:08:42.415867 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:42.415878 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:42.415957 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:42.416124 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:42.416125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:42.416352 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:42.416414 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:42.416416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:42.417139 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:42.417140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:42.417694 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:42.417695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:42.418103 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:42.418104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:42.418151 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:42.418699 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:42.418704 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:42.420940 (+ 2236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
W20260925 10:08:42.421118 23668 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:42.415852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:54177 (local address 127.14.41.126:46043)
0925 10:08:42.415978 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:42.415980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:42.415990 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:08:42.416020 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:42.416020 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:42.416032 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:42.416100 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:42.416495 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:42.417056 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:42.417803 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:42.418018 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:42.418084 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:42.421043 (+ 2959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.65:54177: BlockingRecv error: recv got EOF from 127.14.41.65:54177 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
Sep 25 10:08:42 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:42 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:42.557032 23669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:42.551632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.66:39213)
0925 10:08:42.551884 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:42.551898 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:42.552017 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:42.552186 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:42.552186 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:42.552369 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:42.552418 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:42.552420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:42.553124 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:42.553126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:42.553686 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:42.553688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:42.554103 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:42.554104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:42.554171 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:42.554742 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:42.554747 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:42.556942 (+ 2195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":82,"threads_started":1}
W20260925 10:08:42.557122 23668 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:42.551806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:39213 (local address 127.14.41.126:46043)
0925 10:08:42.551889 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:42.551892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:42.551983 (+ 91us) server_negotiation.cc:407] Connection header received
0925 10:08:42.552092 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:42.552093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:42.552106 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:42.552160 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:42.552509 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:42.553030 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:42.553793 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:42.554011 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:42.554102 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:42.557055 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.66:39213: BlockingRecv error: recv got EOF from 127.14.41.66:39213 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 10:08:42 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:42 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:42.696286 23670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:42.691004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.67:37723)
0925 10:08:42.691193 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:42.691207 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:42.691280 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:42.691451 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:42.691452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:42.691650 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:42.691701 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:42.691703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:42.692602 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:42.692603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:42.693164 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:42.693165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:42.693543 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:42.693544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:42.693589 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:42.694100 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:42.694105 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:42.696190 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":81,"threads_started":1}
W20260925 10:08:42.696362 23668 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:42.691180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:37723 (local address 127.14.41.126:46043)
0925 10:08:42.691278 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:42.691280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:42.691291 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:08:42.691338 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:42.691339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:42.691352 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:42.691408 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:42.691786 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:42.692513 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:42.693275 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:42.693461 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:42.693518 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:42.696312 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.67:37723: BlockingRecv error: recv got EOF from 127.14.41.67:37723 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 10:08:43 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:43 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:43.426608 23671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:43.421599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.65:57157)
0925 10:08:43.421773 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:43.421784 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:43.421840 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:43.421960 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:43.421961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:43.422130 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:43.422182 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:43.422183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:43.422916 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:43.422918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:43.423549 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:43.423552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:43.424104 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:43.424105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:43.424152 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:43.424612 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:43.424617 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:43.426546 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":100,"threads_started":1}
W20260925 10:08:43.426812 23672 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:43.421712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:57157 (local address 127.14.41.126:46043)
0925 10:08:43.421832 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:43.421834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:43.421843 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:08:43.421886 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:43.421887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:43.421898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:43.421955 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:43.422245 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:43.422756 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:43.423674 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:43.424036 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:43.424095 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:43.426684 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.65:57157: BlockingRecv error: recv got EOF from 127.14.41.65:57157 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":51,"threads_started":1}
Sep 25 10:08:43 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:43 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:43.562664 23673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:43.557592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.66:45069)
0925 10:08:43.557829 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:43.557841 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:43.557998 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:43.558185 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:43.558186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:43.558376 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:43.558436 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:43.558438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:43.559133 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:43.559135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:43.559685 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:43.559687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:43.560119 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:43.560120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:43.560180 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:43.560655 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:43.560660 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:43.562601 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":102,"threads_started":1}
W20260925 10:08:43.562827 23672 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:43.557728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:45069 (local address 127.14.41.126:46043)
0925 10:08:43.557806 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:43.557808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:43.557974 (+ 166us) server_negotiation.cc:407] Connection header received
0925 10:08:43.558078 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:43.558079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:43.558098 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:43.558174 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:43.558529 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:43.559044 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:43.559779 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:43.560041 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:43.560121 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:43.562748 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.66:45069: BlockingRecv error: recv got EOF from 127.14.41.66:45069 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 10:08:43 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:43 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:43.702556 23674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:43.696941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.67:56219)
0925 10:08:43.697116 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:43.697132 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:43.697216 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:43.697467 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:43.697468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:43.697643 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:43.697699 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:43.697700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:43.698627 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:43.698628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:43.699383 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:43.699386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:43.699725 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:43.699727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:43.699788 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:43.700476 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:43.700483 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:43.702492 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":74,"threads_started":1}
W20260925 10:08:43.702742 23672 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:43.697084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:56219 (local address 127.14.41.126:46043)
0925 10:08:43.697176 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:43.697178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:43.697202 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:08:43.697336 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:43.697337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:43.697351 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:43.697416 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:43.697883 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:43.698511 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:43.699463 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:43.699647 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:43.699706 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:43.702655 (+ 2949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.67:56219: BlockingRecv error: recv got EOF from 127.14.41.67:56219 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 10:08:44 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:44 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:44.439558 23675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:44.427243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.65:43851)
0925 10:08:44.427412 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:44.427423 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:44.427490 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:44.427638 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:44.427638 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:44.427800 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:44.427859 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:44.427861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:44.428722 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:44.428723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:44.429568 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:44.429571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:44.429989 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:44.429990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:44.430039 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:44.437334 (+ 7295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:44.437342 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:44.439464 (+ 2122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
W20260925 10:08:44.439646 23676 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:44.427350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:43851 (local address 127.14.41.126:46043)
0925 10:08:44.427502 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:44.427503 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:44.427529 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:08:44.427555 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:44.427555 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:44.427567 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:44.427618 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:44.427982 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:44.428624 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:44.429634 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:44.429836 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:44.429972 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:44.439572 (+ 9600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.65:43851: BlockingRecv error: recv got EOF from 127.14.41.65:43851 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
Sep 25 10:08:44 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:44 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:44.568403 23677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:44.563311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.66:58323)
0925 10:08:44.563455 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:44.563470 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:44.563529 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:44.563645 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:44.563646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:44.563906 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:44.564018 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:44.564021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:44.564851 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:44.564852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:44.565502 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:44.565505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:44.565883 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:44.565884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:44.565935 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:44.566417 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:44.566423 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:44.568341 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
W20260925 10:08:44.568542 23676 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:44.563386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:58323 (local address 127.14.41.126:46043)
0925 10:08:44.563469 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:44.563471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:44.563514 (+ 43us) server_negotiation.cc:407] Connection header received
0925 10:08:44.563574 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:44.563575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:44.563586 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:44.563639 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:44.564131 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:44.564754 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:44.565588 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:44.565786 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:44.565848 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:44.568458 (+ 2610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.66:58323: BlockingRecv error: recv got EOF from 127.14.41.66:58323 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 10:08:44 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:44 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:44.708333 23678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:44.703089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.67:41043)
0925 10:08:44.703285 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:44.703299 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:44.703438 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:44.703616 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:44.703617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:44.703808 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:44.703866 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:44.703869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:44.704603 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:44.704605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:44.705137 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:44.705138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:44.705523 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:44.705524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:44.705588 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:44.706104 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:44.706109 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:44.708272 (+ 2163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":100,"threads_started":1}
W20260925 10:08:44.708428 23676 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:44.703214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:41043 (local address 127.14.41.126:46043)
0925 10:08:44.703289 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:44.703292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:44.703416 (+ 124us) server_negotiation.cc:407] Connection header received
0925 10:08:44.703521 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:44.703522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:44.703535 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:44.703610 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:44.704004 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:44.704507 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:44.705233 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:44.705432 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:44.705512 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:44.708377 (+ 2865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.67:41043: BlockingRecv error: recv got EOF from 127.14.41.67:41043 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 10:08:45 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:45 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:45.445525 23679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:45.440189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.65:57855)
0925 10:08:45.440378 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:45.440391 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:45.440448 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:45.440649 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:45.440737 (+ 88us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:45.440926 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:45.440982 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:45.440984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:45.441784 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:45.441785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:45.442319 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:45.442321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:45.442786 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:45.442787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:45.442833 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:45.443306 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:45.443311 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:45.445462 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":103,"threads_started":1}
W20260925 10:08:45.445709 23680 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:45.440306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:57855 (local address 127.14.41.126:46043)
0925 10:08:45.440470 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:45.440472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:45.440482 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:08:45.440505 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:45.440506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:45.440523 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:45.440587 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:45.441092 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:45.441628 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:45.442418 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:45.442624 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:45.442686 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:45.445613 (+ 2927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.65:57855: BlockingRecv error: recv got EOF from 127.14.41.65:57855 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
Sep 25 10:08:45 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:45 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:45.574544 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:45.568997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.66:48277)
0925 10:08:45.569135 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:45.569151 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:45.569235 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:45.569504 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:45.569506 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:45.569671 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:45.569721 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:45.569722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:45.570557 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:45.570559 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:45.571143 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:45.571145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:45.571821 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:45.571822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:45.571877 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:45.572453 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:45.572458 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:45.574483 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
W20260925 10:08:45.574755 23680 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:45.569151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:48277 (local address 127.14.41.126:46043)
0925 10:08:45.569228 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:45.569231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:45.569243 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:08:45.569315 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:45.569316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:45.569329 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:45.569447 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:45.569856 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:45.570404 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:45.571223 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:45.571603 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:45.571799 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:45.574662 (+ 2863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.66:48277: BlockingRecv error: recv got EOF from 127.14.41.66:48277 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 10:08:45 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:45 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:45.714020 23682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:45.708912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.67:50617)
0925 10:08:45.709044 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:45.709053 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:45.709095 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:45.709282 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:45.709283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:45.709449 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:45.709499 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:45.709501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:45.710257 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:45.710259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:45.710796 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:45.710797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:45.711145 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:45.711146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:45.711185 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:45.711626 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:45.711631 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:45.713931 (+ 2300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":54,"threads_started":1}
W20260925 10:08:45.714156 23680 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:45.709020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:50617 (local address 127.14.41.126:46043)
0925 10:08:45.709097 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:45.709098 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:45.709139 (+ 41us) server_negotiation.cc:407] Connection header received
0925 10:08:45.709161 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:45.709162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:45.709174 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:45.709219 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:45.709612 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:45.710165 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:45.710901 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:45.711079 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:45.711131 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:45.714069 (+ 2938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.67:50617: BlockingRecv error: recv got EOF from 127.14.41.67:50617 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 10:08:46 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:46 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:46.450961 23683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:46.445998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.65:58771)
0925 10:08:46.446240 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:46.446255 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:46.446329 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:46.446546 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:46.446547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:46.446729 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:46.446784 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:46.446785 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:46.447460 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:46.447462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:46.448029 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:46.448031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:46.448384 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:46.448385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:46.448427 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:46.448960 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:46.448966 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:46.450876 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":121,"threads_started":1}
W20260925 10:08:46.451118 23684 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:46.446080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:58771 (local address 127.14.41.126:46043)
0925 10:08:46.446204 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:46.446206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:46.446367 (+ 161us) server_negotiation.cc:407] Connection header received
0925 10:08:46.446401 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:46.446402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:46.446416 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:46.446476 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:46.446860 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:46.447362 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:46.448107 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:46.448305 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:46.448360 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:46.451018 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.65:58771: BlockingRecv error: recv got EOF from 127.14.41.65:58771 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":51,"threads_started":1}
Sep 25 10:08:46 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:46 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:46.580591 23685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:46.575114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.66:50621)
0925 10:08:46.575364 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:46.575376 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:46.575442 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:46.575678 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:46.575679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:46.575917 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:46.575988 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:46.575991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:46.576884 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:46.576885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:46.577427 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:46.577428 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:46.577773 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:46.577774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:46.577819 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:46.578291 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:46.578295 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:46.580530 (+ 2235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":174,"threads_started":1}
W20260925 10:08:46.580878 23684 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:46.575238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:50621 (local address 127.14.41.126:46043)
0925 10:08:46.575367 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:46.575370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:46.575442 (+ 72us) server_negotiation.cc:407] Connection header received
0925 10:08:46.575496 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:46.575497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:46.575508 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:46.575659 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:46.576050 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:46.576795 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:46.577510 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:46.577702 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:46.577760 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:46.580801 (+ 3041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.66:50621: BlockingRecv error: recv got EOF from 127.14.41.66:50621 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 25 10:08:46 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:46 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:46.719970 23686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:46.714807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.67:37623)
0925 10:08:46.714952 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:46.714968 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:46.715029 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:46.715177 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:46.715178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:46.715351 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:46.715402 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:46.715403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:46.716211 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:46.716212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:46.716752 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:46.716754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:46.717117 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:46.717118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:46.717168 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:46.717703 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:46.717708 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:46.719831 (+ 2123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
W20260925 10:08:46.720091 23684 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:46.714893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:37623 (local address 127.14.41.126:46043)
0925 10:08:46.714967 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:46.714969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:46.715036 (+ 67us) server_negotiation.cc:407] Connection header received
0925 10:08:46.715086 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:46.715087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:46.715100 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:46.715144 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:46.715481 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:46.716067 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:46.716830 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:46.717045 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:46.717098 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:46.720019 (+ 2921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.67:37623: BlockingRecv error: recv got EOF from 127.14.41.67:37623 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 10:08:47 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:47 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:47.456903 23687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:47.451535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.65:57759)
0925 10:08:47.451698 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:47.451713 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:47.451763 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:47.452126 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:47.452127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:47.452298 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:47.452385 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:47.452386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:47.453120 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:47.453121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:47.453693 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:47.453696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:47.454084 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:47.454085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:47.454134 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:47.454662 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:47.454667 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:47.456801 (+ 2134us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
W20260925 10:08:47.456964 23688 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:47.451606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:57759 (local address 127.14.41.126:46043)
0925 10:08:47.451750 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:47.451752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:47.451763 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:08:47.451992 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:47.451993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:47.452008 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:47.452078 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:47.452473 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:47.453041 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:47.453793 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:47.453998 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:47.454061 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:47.456888 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.65:57759: BlockingRecv error: recv got EOF from 127.14.41.65:57759 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
Sep 25 10:08:47 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:47 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:47.586464 23689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:47.581269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.66:43665)
0925 10:08:47.581448 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:47.581462 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:47.581539 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:47.581688 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:47.581689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:47.581860 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:47.581911 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:47.581913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:47.582575 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:47.582576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:47.583173 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:47.583176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:47.583605 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:47.583607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:47.583704 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:47.584326 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:47.584332 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:47.586402 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":58,"threads_started":1}
W20260925 10:08:47.586623 23688 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:47.581445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:43665 (local address 127.14.41.126:46043)
0925 10:08:47.581518 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:47.581520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:47.581531 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:08:47.581599 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:47.581600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:47.581612 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:47.581662 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:47.581976 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:47.582500 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:47.583258 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:47.583485 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:47.583543 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:47.586529 (+ 2986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.66:43665: BlockingRecv error: recv got EOF from 127.14.41.66:43665 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 10:08:47 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:47 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:47.726580 23690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:47.720530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.67:48017)
0925 10:08:47.720674 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:47.720686 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:47.720766 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:47.720950 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:47.720951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:47.721200 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:47.721270 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:47.721273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:47.722412 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:47.722414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:47.723258 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:47.723261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:47.723764 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:47.723765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:47.723834 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:47.724375 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:47.724380 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:47.726469 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
W20260925 10:08:47.726737 23688 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:47.720597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:48017 (local address 127.14.41.126:46043)
0925 10:08:47.720658 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:47.720660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:47.720744 (+ 84us) server_negotiation.cc:407] Connection header received
0925 10:08:47.720827 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:47.720828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:47.720844 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:47.720929 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:47.721356 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:47.722287 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:47.723359 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:47.723656 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:47.723752 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:47.726654 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.67:48017: BlockingRecv error: recv got EOF from 127.14.41.67:48017 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 10:08:48 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:48 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:48.462720 23691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:48.457591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.65:44059)
0925 10:08:48.457764 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:48.457779 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:48.457848 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:48.458018 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:48.458019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:48.458184 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:48.458237 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:48.458239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:48.459041 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:48.459043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:48.459607 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:48.459609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:48.460163 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:48.460164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:48.460217 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:48.460793 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:48.460798 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:48.462651 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":74,"threads_started":1}
W20260925 10:08:48.462966 23692 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:48.457747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:44059 (local address 127.14.41.126:46043)
0925 10:08:48.457892 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:48.457894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:48.457901 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:08:48.457923 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:48.457924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:48.457934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:48.457992 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:48.458318 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:48.458880 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:48.459799 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:48.460090 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:48.460146 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:48.462894 (+ 2748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.65:44059: BlockingRecv error: recv got EOF from 127.14.41.65:44059 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
Sep 25 10:08:48 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:48 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:48.593089 23693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:48.587107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.66:47267)
0925 10:08:48.587349 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:48.587366 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:48.587475 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:48.587654 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:48.587656 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:48.587952 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:48.588005 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:48.588008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:48.588882 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:48.588883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:48.589448 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:48.589450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:48.589845 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:48.589846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:48.589904 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:48.590445 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:48.590451 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:48.592945 (+ 2494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":110,"threads_started":1}
W20260925 10:08:48.593127 23692 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:48.587220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:47267 (local address 127.14.41.126:46043)
0925 10:08:48.587379 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:48.587381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:48.587474 (+ 93us) server_negotiation.cc:407] Connection header received
0925 10:08:48.587546 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:48.587547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:48.587561 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:48.587650 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:48.588106 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:48.588783 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:48.589552 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:48.589759 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:48.589844 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:48.593047 (+ 3203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.66:47267: BlockingRecv error: recv got EOF from 127.14.41.66:47267 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 10:08:48 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:48 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:48.733222 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:48.727282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.67:56257)
0925 10:08:48.727457 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:48.727472 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:48.727541 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:48.727699 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:48.727700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:48.727929 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:48.728034 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:48.728036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:48.729103 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:48.729104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:48.729671 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:48.729673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:48.730131 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:48.730132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:48.730181 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:48.730775 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:48.730781 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:48.733146 (+ 2365us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":158,"threads_started":1}
W20260925 10:08:48.733407 23692 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:48.727467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:56257 (local address 127.14.41.126:46043)
0925 10:08:48.727534 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:48.727540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:48.727553 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:08:48.727585 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:48.727586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:48.727598 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:48.727667 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:48.728169 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:48.729016 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:48.729851 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:48.730054 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:48.730116 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:48.733328 (+ 3212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.67:56257: BlockingRecv error: recv got EOF from 127.14.41.67:56257 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 10:08:49 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:49 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:49.477005 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:49.463513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.65:51933)
0925 10:08:49.463665 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:49.463691 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:49.463758 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:49.464035 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:49.464036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:49.464298 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:49.464376 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:49.464378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:49.465170 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:49.465171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:49.465767 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:49.465769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:49.466179 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:49.466180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:49.466241 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:49.474669 (+ 8428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:49.474677 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:49.476925 (+ 2248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
W20260925 10:08:49.477322 23696 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:49.463628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:51933 (local address 127.14.41.126:46043)
0925 10:08:49.463757 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:49.463759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:49.463805 (+ 46us) server_negotiation.cc:407] Connection header received
0925 10:08:49.463829 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:49.463830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:49.463841 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:49.463977 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:49.464468 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:49.465083 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:49.465866 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:49.466078 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:49.466138 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:49.477155 (+ 11017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.65:51933: BlockingRecv error: recv got EOF from 127.14.41.65:51933 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
Sep 25 10:08:49 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:49 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:49.599773 23697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:49.593739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.66:47723)
0925 10:08:49.594005 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:49.594020 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:49.594144 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:49.594405 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:49.594406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:49.594623 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:49.594712 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:49.594715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:49.595663 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:49.595664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:49.596289 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:49.596291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:49.596821 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:49.596822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:49.596876 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:49.597391 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:49.597396 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:49.599689 (+ 2293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":191,"threads_started":1}
W20260925 10:08:49.600018 23696 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:49.593871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:47723 (local address 127.14.41.126:46043)
0925 10:08:49.594004 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:49.594007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:49.594113 (+ 106us) server_negotiation.cc:407] Connection header received
0925 10:08:49.594242 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:49.594244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:49.594258 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:49.594323 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:49.594825 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:49.595580 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:49.596406 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:49.596627 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:49.596698 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:49.599931 (+ 3233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.66:47723: BlockingRecv error: recv got EOF from 127.14.41.66:47723 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 25 10:08:49 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:49 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:49.740097 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:49.733910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.67:32863)
0925 10:08:49.734055 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:49.734067 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:49.734139 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:49.734334 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:49.734335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:49.734562 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:49.734616 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:49.734618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:49.735362 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:49.735363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:49.736184 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:49.736188 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:49.736862 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:49.736864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:49.736916 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:49.737605 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:49.737610 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:49.740035 (+ 2425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
W20260925 10:08:49.740268 23696 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:49.733985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:32863 (local address 127.14.41.126:46043)
0925 10:08:49.734063 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:49.734065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:49.734193 (+ 128us) server_negotiation.cc:407] Connection header received
0925 10:08:49.734227 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:49.734228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:49.734247 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:49.734304 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:49.734703 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:49.735252 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:49.736311 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:49.736724 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:49.736816 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:49.740192 (+ 3376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.67:32863: BlockingRecv error: recv got EOF from 127.14.41.67:32863 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 10:08:50 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:50 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:50.483453 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:50.477635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.65:52993)
0925 10:08:50.477923 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:50.477936 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:50.478015 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:50.478186 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:50.478187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:50.478363 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:50.478416 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:50.478417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:50.479301 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:50.479303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:50.480128 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:50.480132 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:50.480483 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:50.480484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:50.480525 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:50.481056 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:50.481061 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:50.483369 (+ 2308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":202,"threads_started":1}
W20260925 10:08:50.483582 23700 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:50.477838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:52993 (local address 127.14.41.126:46043)
0925 10:08:50.477961 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:50.477963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:50.478063 (+ 100us) server_negotiation.cc:407] Connection header received
0925 10:08:50.478092 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:50.478094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:50.478104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:50.478167 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:50.478495 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:50.479142 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:50.480204 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:50.480414 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:50.480474 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:50.483465 (+ 2991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.65:52993: BlockingRecv error: recv got EOF from 127.14.41.65:52993 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
Sep 25 10:08:50 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:50 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:50.606189 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:50.600318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.66:42025)
0925 10:08:50.600593 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:50.600606 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:50.600722 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:50.601006 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:50.601007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:50.601220 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:50.601273 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:50.601275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:50.602028 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:50.602030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:50.602663 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:50.602665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:50.603133 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:50.603134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:50.603206 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:50.603726 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:50.603732 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:50.606062 (+ 2330us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":149,"threads_started":1}
W20260925 10:08:50.606235 23700 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:50.600455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:42025 (local address 127.14.41.126:46043)
0925 10:08:50.600584 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:50.600587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:50.600707 (+ 120us) server_negotiation.cc:407] Connection header received
0925 10:08:50.600901 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:50.600903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:50.600919 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:50.600999 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:50.601368 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:50.601924 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:50.602789 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:50.603027 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:50.603132 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:50.606168 (+ 3036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.66:42025: BlockingRecv error: recv got EOF from 127.14.41.66:42025 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 25 10:08:50 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:50 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:50.745805 23702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:50.740598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.67:51367)
0925 10:08:50.740769 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:50.740784 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:50.740896 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:50.741044 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:50.741045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:50.741218 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:50.741268 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:50.741270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:50.742071 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:50.742072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:50.742604 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:50.742606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:50.743039 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:50.743039 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:50.743080 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:50.743634 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:50.743639 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:50.745691 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":108,"threads_started":1}
W20260925 10:08:50.745853 23700 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:50.740749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:51367 (local address 127.14.41.126:46043)
0925 10:08:50.740800 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:50.740802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:50.740884 (+ 82us) server_negotiation.cc:407] Connection header received
0925 10:08:50.740960 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:50.740961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:50.740974 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:50.741017 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:50.741392 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:50.741982 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:50.742704 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:50.742957 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:50.743027 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:50.745783 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.67:51367: BlockingRecv error: recv got EOF from 127.14.41.67:51367 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 10:08:51 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:51 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:51.489714 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:51.484128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.65:58347)
0925 10:08:51.484383 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:51.484396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:51.484460 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:51.484648 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:51.484650 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:51.484824 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:51.484885 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:51.484887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:51.485661 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:51.485662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:51.486324 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:51.486327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:51.486716 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:51.486716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:51.486760 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:51.487366 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:51.487371 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:51.489603 (+ 2232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":116,"threads_started":1}
W20260925 10:08:51.489785 23704 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:51.484323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:58347 (local address 127.14.41.126:46043)
0925 10:08:51.484456 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:51.484458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:51.484469 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:08:51.484517 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:51.484518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:51.484529 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:51.484591 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:51.485007 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:51.485559 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:51.486429 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:51.486638 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:51.486705 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:51.489703 (+ 2998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.65:58347: BlockingRecv error: recv got EOF from 127.14.41.65:58347 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
Sep 25 10:08:51 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:51 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:51.611908 23705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:51.606792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.66:37755)
0925 10:08:51.607000 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:51.607013 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:51.607109 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:51.607271 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:51.607272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:51.607475 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:51.607525 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:51.607527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:51.608262 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:51.608263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:51.608837 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:51.608839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:51.609358 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:51.609359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:51.609436 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:51.609946 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:51.609951 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:51.611835 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":108,"threads_started":1}
W20260925 10:08:51.612131 23704 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:51.606902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:37755 (local address 127.14.41.126:46043)
0925 10:08:51.606984 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:51.606986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:51.607089 (+ 103us) server_negotiation.cc:407] Connection header received
0925 10:08:51.607172 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:51.607173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:51.607187 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:51.607263 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:51.607618 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:51.608179 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:51.608933 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:51.609132 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:51.609357 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:51.611987 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.66:37755: BlockingRecv error: recv got EOF from 127.14.41.66:37755 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 10:08:51 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:51 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:51.751804 23706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:51.746567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.67:34165)
0925 10:08:51.746695 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:51.746705 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:51.746750 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:51.746950 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:51.746951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:51.747139 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:51.747188 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:51.747190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:51.748146 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:51.748147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:51.748867 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:51.748870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:51.749288 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:51.749289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:51.749375 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:51.749873 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:51.749879 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:51.751736 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":56,"threads_started":1}
W20260925 10:08:51.751936 23704 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:51.746691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:34165 (local address 127.14.41.126:46043)
0925 10:08:51.746740 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:51.746742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:51.746757 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:08:51.746803 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:51.746803 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:51.746817 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:51.746886 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:51.747277 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:51.748005 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:51.748991 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:51.749207 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:51.749281 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:51.751849 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.67:34165: BlockingRecv error: recv got EOF from 127.14.41.67:34165 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 10:08:52 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:52 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:52.495822 23707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:52.490432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.65:35453)
0925 10:08:52.490590 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:52.490602 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:52.490664 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:52.490902 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:52.490904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:52.491077 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:52.491126 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:52.491127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:52.491897 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:52.491898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:52.492489 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:52.492491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:52.492872 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:52.492874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:52.492936 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:52.493427 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:52.493432 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:52.495761 (+ 2329us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":98,"threads_started":1}
W20260925 10:08:52.495950 23708 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:52.490587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:35453 (local address 127.14.41.126:46043)
0925 10:08:52.490724 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:52.490727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:52.490737 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:08:52.490772 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:52.490772 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:52.490784 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:52.490847 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:52.491235 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:52.491809 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:52.492571 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:52.492776 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:52.492833 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:52.495862 (+ 3029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.65:35453: BlockingRecv error: recv got EOF from 127.14.41.65:35453 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
Sep 25 10:08:52 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:52 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:52.617233 23709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:52.612489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.66:57171)
0925 10:08:52.612683 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:52.612695 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:52.612762 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:52.612913 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:52.612914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:52.613083 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:52.613132 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:52.613133 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:52.613757 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:52.613758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:52.614291 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:52.614292 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:52.614622 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:52.614623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:52.614665 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:52.615200 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:52.615211 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:52.617124 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":123,"threads_started":1}
W20260925 10:08:52.617301 23708 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:52.612595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:57171 (local address 127.14.41.126:46043)
0925 10:08:52.612669 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:52.612671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:52.612763 (+ 92us) server_negotiation.cc:407] Connection header received
0925 10:08:52.612826 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:52.612827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:52.612838 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:52.612887 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:52.613196 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:52.613685 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:52.614371 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:52.614555 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:52.614606 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:52.617228 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.66:57171: BlockingRecv error: recv got EOF from 127.14.41.66:57171 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 10:08:52 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:52 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:52.757608 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:52.752404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.67:39157)
0925 10:08:52.752562 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:52.752579 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:52.752650 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:52.752787 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:52.752788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:52.753008 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:52.753054 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:52.753056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:52.753712 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:52.753713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:52.754273 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:52.754275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:52.754769 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:52.754771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:52.754823 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:52.755315 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:52.755319 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:52.757528 (+ 2209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
W20260925 10:08:52.757660 23708 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:52.752468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:39157 (local address 127.14.41.126:46043)
0925 10:08:52.752519 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:52.752521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:52.752651 (+ 130us) server_negotiation.cc:407] Connection header received
0925 10:08:52.752698 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:52.752699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:52.752711 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:52.752767 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:52.753131 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:52.753632 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:52.754373 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:52.754563 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:52.754622 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:52.757608 (+ 2986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.67:39157: BlockingRecv error: recv got EOF from 127.14.41.67:39157 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 10:08:53 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:53 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:53.501905 23711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:53.496397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.65:46501)
0925 10:08:53.496551 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:53.496561 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:53.496613 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:53.496853 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:53.496854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:53.497031 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:53.497078 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:53.497079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:53.497905 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:53.497906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:53.498475 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:53.498477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:53.499201 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:53.499202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:53.499257 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:53.499857 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:53.499863 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:53.501826 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
W20260925 10:08:53.502087 23712 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:53.496456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:46501 (local address 127.14.41.126:46043)
0925 10:08:53.496616 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:53.496618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:53.496629 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:08:53.496658 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:53.496658 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:53.496669 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:53.496844 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:53.497158 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:53.497817 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:53.498573 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:53.498955 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:53.499035 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:53.501989 (+ 2954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.65:46501: BlockingRecv error: recv got EOF from 127.14.41.65:46501 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
Sep 25 10:08:53 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:53 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:53.623454 23713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:53.617921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.66:52401)
0925 10:08:53.618088 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:53.618103 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:53.618168 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:53.618304 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:53.618305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:53.618481 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:53.618528 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:53.618529 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:53.619348 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:53.619349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:53.619905 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:53.619907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:53.620348 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:53.620349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:53.620401 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:53.620994 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:53.620999 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:53.623336 (+ 2337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
W20260925 10:08:53.623507 23712 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:53.618077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:52401 (local address 127.14.41.126:46043)
0925 10:08:53.618149 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:53.618151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:53.618160 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:08:53.618232 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:53.618233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:53.618244 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:53.618293 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:53.618627 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:53.619263 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:53.620043 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:53.620251 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:53.620316 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:53.623436 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.66:52401: BlockingRecv error: recv got EOF from 127.14.41.66:52401 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 10:08:53 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:53 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:53.764042 23714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:53.758241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.67:55837)
0925 10:08:53.758453 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:53.758472 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:53.758544 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:53.758695 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:53.758696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:53.758876 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:53.758930 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:53.758931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:53.759980 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:53.759982 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:53.760710 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:53.760713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:53.761090 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:53.761091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:53.761133 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:53.761637 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:53.761644 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:53.763905 (+ 2261us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":91,"threads_started":1}
W20260925 10:08:53.764098 23712 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:53.758390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:55837 (local address 127.14.41.126:46043)
0925 10:08:53.758465 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:53.758468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:53.758539 (+ 71us) server_negotiation.cc:407] Connection header received
0925 10:08:53.758601 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:53.758602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:53.758616 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:53.758667 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:53.759036 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:53.759792 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:53.760816 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:53.761010 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:53.761079 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:53.764024 (+ 2945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.67:55837: BlockingRecv error: recv got EOF from 127.14.41.67:55837 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 10:08:54 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:54 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:54.511832 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:54.502433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.65:54277)
0925 10:08:54.502591 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:54.502610 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:54.502663 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:54.502814 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:54.502814 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:54.503028 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:54.503077 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:54.503079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:54.503755 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:54.503757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:54.504377 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:54.504381 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:54.504752 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:54.504753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:54.504807 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:54.509627 (+ 4820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:54.509633 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:54.511762 (+ 2129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
W20260925 10:08:54.512086 23716 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:54.502528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:54277 (local address 127.14.41.126:46043)
0925 10:08:54.502663 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:54.502665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:54.502701 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:08:54.502730 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:54.502731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:54.502741 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:54.502793 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:54.503153 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:54.503656 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:54.504460 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:54.504656 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:54.504713 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:54.511997 (+ 7284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.65:54277: BlockingRecv error: recv got EOF from 127.14.41.65:54277 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
Sep 25 10:08:54 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:54 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:54.628924 23717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:54.623978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.66:41929)
0925 10:08:54.624158 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:54.624172 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:54.624245 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:54.624451 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:54.624452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:54.624682 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:54.624733 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:54.624735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:54.625408 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:54.625409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:54.625957 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:54.625959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:54.626295 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:54.626297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:54.626347 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:54.626802 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:54.626807 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:54.628843 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":77,"threads_started":1}
W20260925 10:08:54.629124 23716 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:54.624228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:41929 (local address 127.14.41.126:46043)
0925 10:08:54.624317 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:54.624319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:54.624328 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:08:54.624356 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:54.624357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:54.624368 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:54.624417 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:54.624790 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:54.625327 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:54.626041 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:54.626227 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:54.626280 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:54.629070 (+ 2790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.66:41929: BlockingRecv error: recv got EOF from 127.14.41.66:41929 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 10:08:54 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:54 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.67@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:54.770004 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:54.764769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.67:49515)
0925 10:08:54.764971 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:54.764983 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:54.765075 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:54.765238 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:54.765239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:54.765464 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:54.765515 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:54.765517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:54.766297 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:54.766298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:54.766865 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:54.766867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:54.767260 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:54.767262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:54.767330 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:54.767806 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:54.767812 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:54.769913 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":86,"threads_started":1}
W20260925 10:08:54.770040 23716 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:54.764887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:49515 (local address 127.14.41.126:46043)
0925 10:08:54.764964 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:54.764966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:54.765055 (+ 89us) server_negotiation.cc:407] Connection header received
0925 10:08:54.765145 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:54.765146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:54.765161 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:54.765233 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:54.765621 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:54.766190 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:54.766964 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:54.767162 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:54.767260 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:54.769976 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.67:49515: BlockingRecv error: recv got EOF from 127.14.41.67:49515 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 10:08:55 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:55 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.65@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:55.518231 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:55.512479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.65:37839)
0925 10:08:55.512651 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:55.512664 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:55.512756 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:55.512933 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:55.512933 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:55.513098 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:55.513187 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:55.513189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:55.514095 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:55.514096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:55.514791 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:55.514793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:55.515189 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:55.515189 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:55.515236 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:55.516050 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:55.516055 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:55.518162 (+ 2107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":104,"threads_started":1}
W20260925 10:08:55.518390 23720 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:55.512590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:37839 (local address 127.14.41.126:46043)
0925 10:08:55.512779 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:55.512787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:55.512797 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:08:55.512824 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:55.512825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:55.512837 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:55.512911 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:55.513272 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:55.513961 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:55.514873 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:55.515093 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:55.515156 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:55.518297 (+ 3141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.65:37839: BlockingRecv error: recv got EOF from 127.14.41.65:37839 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":127,"threads_started":1}
Sep 25 10:08:55 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:08:55 dist-test-slave-s6tm krb5kdc[22661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.14.41.66@KRBTEST.COM for oryx/127.14.41.126@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:08:55.634622 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:55.629528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46043 (local address 127.14.41.66:37793)
0925 10:08:55.629736 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:55.629755 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:55.629830 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:55.630011 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:55.630012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:55.630185 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:55.630237 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:55.630239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:55.630959 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:55.630960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:55.631579 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:55.631581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:55.632041 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:55.632042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:55.632096 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:55.632566 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:55.632570 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:55.634546 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.14.41.126:46043: Server oryx/127.14.41.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":107,"threads_started":1}
W20260925 10:08:55.634756 23720 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:55.629680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:37793 (local address 127.14.41.126:46043)
0925 10:08:55.629778 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:55.629780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:55.629813 (+ 33us) server_negotiation.cc:407] Connection header received
0925 10:08:55.629905 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:55.629906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:55.629918 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:55.629983 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:55.630341 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:55.630876 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:55.631668 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:55.631872 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:55.632026 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:55.634650 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.14.41.66:37793: BlockingRecv error: recv got EOF from 127.14.41.66:37793 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:08:55.680464 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 23244
I20260925 10:08:55.685415 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 23396
I20260925 10:08:55.690263 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 23530
I20260925 10:08:55.694942 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 23174
2026-09-25T10:08:55Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17589 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T10:08:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:55Z Disabled control of system clock
I20260925 10:08:55.727581 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:37469
--webserver_interface=127.14.41.126
--webserver_port=0
--builtin_ntp_servers=127.14.41.84:43673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:37469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 10:08:55.812067 23725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:55.812281 23725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:55.812304 23725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:55.813476 23725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:55.813517 23725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:55.813529 23725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:55.813544 23725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:55.813555 23725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:08:55.814873 23725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:43673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:37469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:37469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:55.815085 23725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:55.815359 23725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:55.820834 23731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:55.820958 23733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:55.820878 23725 server_base.cc:1034] running on GCE node
W20260925 10:08:55.820922 23730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:55.821275 23725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:55.821619 23725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:55.822809 23725 hybrid_clock.cc:648] HybridClock initialized: now 1790330935822795 us; error 54 us; skew 500 ppm
I20260925 10:08:55.824173 23725 webserver.cc:460] Webserver started at http://127.14.41.126:37355/ using document root <none> and password file <none>
I20260925 10:08:55.824483 23725 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:55.824534 23725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:55.824712 23725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:55.825842 23725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "f1fedd596e3d4ab9a1d929aed0ca3009"
format_stamp: "Formatted at 2026-09-25 10:08:55 on dist-test-slave-s6tm"
server_key: "03285f0cd77480a58965cb4b418694e7"
server_key_iv: "79e8ffb6c00492f5d84df8a41bc4ac49"
server_key_version: "encryptionkey@0"
I20260925 10:08:55.826215 23725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "f1fedd596e3d4ab9a1d929aed0ca3009"
format_stamp: "Formatted at 2026-09-25 10:08:55 on dist-test-slave-s6tm"
server_key: "03285f0cd77480a58965cb4b418694e7"
server_key_iv: "79e8ffb6c00492f5d84df8a41bc4ac49"
server_key_version: "encryptionkey@0"
I20260925 10:08:55.827518 23725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:08:55.828323 23739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:55.828495 23725 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:55.828580 23725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "f1fedd596e3d4ab9a1d929aed0ca3009"
format_stamp: "Formatted at 2026-09-25 10:08:55 on dist-test-slave-s6tm"
server_key: "03285f0cd77480a58965cb4b418694e7"
server_key_iv: "79e8ffb6c00492f5d84df8a41bc4ac49"
server_key_version: "encryptionkey@0"
I20260925 10:08:55.828636 23725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:55.859153 23725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:55.859465 23725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:55.859576 23725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:55.863876 23725 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:37469
I20260925 10:08:55.864158 23791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:37469 every 8 connection(s)
I20260925 10:08:55.864344 23725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:55.865060 23792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:55.867816 23792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009: Bootstrap starting.
I20260925 10:08:55.868482 23792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:55.868793 23792 log.cc:824] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:55.869633 23792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009: No bootstrap required, opened a new log
I20260925 10:08:55.871551 23792 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1fedd596e3d4ab9a1d929aed0ca3009" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37469 } }
I20260925 10:08:55.871749 23792 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:55.871806 23792 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1fedd596e3d4ab9a1d929aed0ca3009, State: Initialized, Role: FOLLOWER
I20260925 10:08:55.872032 23792 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1fedd596e3d4ab9a1d929aed0ca3009" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37469 } }
I20260925 10:08:55.872157 23792 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:08:55.872206 23792 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:55.872287 23792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:55.872766 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 23725
I20260925 10:08:55.872925 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:08:55.873008 23792 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1fedd596e3d4ab9a1d929aed0ca3009" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37469 } }
I20260925 10:08:55.873140 14501 external_mini_cluster.cc:1428] Setting key 29027526fd5eaa8fa34fe1616bacbecd
I20260925 10:08:55.873149 23792 leader_election.cc:302] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1fedd596e3d4ab9a1d929aed0ca3009; no voters:
I20260925 10:08:55.873338 23792 leader_election.cc:288] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:08:55.873436 23795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:55.873626 23795 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009 [term 1 LEADER]: Becoming Leader. State: Replica: f1fedd596e3d4ab9a1d929aed0ca3009, State: Running, Role: LEADER
I20260925 10:08:55.873643 23792 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:55.873790 23795 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1fedd596e3d4ab9a1d929aed0ca3009" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37469 } }
I20260925 10:08:55.874320 23796 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1fedd596e3d4ab9a1d929aed0ca3009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1fedd596e3d4ab9a1d929aed0ca3009" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37469 } } }
I20260925 10:08:55.874404 23796 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:55.874464 23797 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1fedd596e3d4ab9a1d929aed0ca3009. Latest consensus state: current_term: 1 leader_uuid: "f1fedd596e3d4ab9a1d929aed0ca3009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1fedd596e3d4ab9a1d929aed0ca3009" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37469 } } }
I20260925 10:08:55.874557 23797 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:55.874923 23806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:55.875438 23806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:55.878787 23806 catalog_manager.cc:1347] Generated new cluster ID: b4f6965bcf8b432a8a154d692645fbad
I20260925 10:08:55.878859 23806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:08:55.881775 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:55.873842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50858 (local address 127.14.41.126:37469)
0925 10:08:55.876183 (+ 2341us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:55.876192 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:55.876226 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:08:55.876348 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:55.876355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:55.876425 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:55.876833 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:55.876926 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:55.878262 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:55.879422 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:55.880006 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:55.880130 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:55.881044 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:55.881048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:55.881049 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:55.881240 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:55.881285 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:55.881297 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:55.881378 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:55.881453 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 10:08:55.881537 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2266,"thread_start_us":88,"threads_started":1}
I20260925 10:08:55.896703 23806 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:08:55.897324 23806 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:55.908360 23806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1fedd596e3d4ab9a1d929aed0ca3009: Generated new TSK 0
I20260925 10:08:55.908617 23806 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:08:55.912938 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37469
--builtin_ntp_servers=127.14.41.84:43673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 10:08:55.997134 23816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:55.997399 23816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:55.997437 23816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:55.998682 23816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:55.998767 23816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:55.998816 23816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:08:56.000326 23816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:43673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:56.000629 23816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:56.000950 23816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:56.006290 23824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:56.006280 23821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:56.006570 23816 server_base.cc:1034] running on GCE node
W20260925 10:08:56.006289 23822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:56.006871 23816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:56.007190 23816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:56.008569 23816 hybrid_clock.cc:648] HybridClock initialized: now 1790330936008530 us; error 56 us; skew 500 ppm
I20260925 10:08:56.010145 23816 webserver.cc:460] Webserver started at http://127.14.41.65:39961/ using document root <none> and password file <none>
I20260925 10:08:56.010430 23816 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:56.010509 23816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:56.010664 23816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:56.012156 23816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "0fc693854acc4738b0761c0058238925"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "ad094a8c44d5e1a3617c070240878081"
server_key_iv: "4e41c06f4553bb5acd3c81f0f656f59d"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.012602 23816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "0fc693854acc4738b0761c0058238925"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "ad094a8c44d5e1a3617c070240878081"
server_key_iv: "4e41c06f4553bb5acd3c81f0f656f59d"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.014130 23816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:08:56.015039 23830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:56.015386 23816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:56.015524 23816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "0fc693854acc4738b0761c0058238925"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "ad094a8c44d5e1a3617c070240878081"
server_key_iv: "4e41c06f4553bb5acd3c81f0f656f59d"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.015615 23816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:56.035302 23816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:56.035686 23816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:56.035862 23816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:56.036150 23816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:56.036514 23816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:56.036577 23816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:56.036623 23816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:56.036661 23816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:56.042645 23816 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:34411
I20260925 10:08:56.042726 23943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:34411 every 8 connection(s)
I20260925 10:08:56.043113 23816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:08:56.048447 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 23816
I20260925 10:08:56.048574 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:08:56.048827 14501 external_mini_cluster.cc:1428] Setting key 872360a66effcb894b562d286aadaaab
I20260925 10:08:56.049068 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:56.043989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:58915 (local address 127.14.41.126:37469)
0925 10:08:56.044041 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:56.044043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:56.044260 (+ 217us) server_negotiation.cc:407] Connection header received
0925 10:08:56.044776 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:56.044778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:56.044791 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:56.044852 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:56.045565 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:56.046070 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:56.047137 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:56.047328 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:56.047421 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:56.048744 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:56.048747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:56.048747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:56.048755 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:56.048774 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:56.048778 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:56.048840 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:56.048943 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 10:08:56.048986 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:08:56.049345 23946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:56.043867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:37469 (local address 127.14.41.65:58915)
0925 10:08:56.044122 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:56.044166 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:56.044665 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:56.044918 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:56.044921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:56.044930 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:56.045476 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:56.045482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:56.046170 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:56.046171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:56.047056 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:56.047059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:56.047447 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:56.047448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:56.047561 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:56.048490 (+ 929us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:08:56.048491 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:56.048523 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:08:56.048548 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:56.048841 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:56.048848 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:56.048851 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:56.048939 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 10:08:56.049062 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":150,"threads_started":1}
I20260925 10:08:56.049871 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37469
--builtin_ntp_servers=127.14.41.84:43673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 10:08:56.049882 23944 heartbeater.cc:347] Connected to a master server at 127.14.41.126:37469
I20260925 10:08:56.050001 23944 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:56.050195 23944 heartbeater.cc:511] Master 127.14.41.126:37469 requested a full tablet report, sending...
I20260925 10:08:56.050757 23756 ts_manager.cc:194] Registered new tserver with Master: 0fc693854acc4738b0761c0058238925 (127.14.41.65:34411)
I20260925 10:08:56.051398 23756 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.41.65:58915
W20260925 10:08:56.135149 23947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:56.135396 23947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:56.135433 23947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:56.136626 23947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:56.136703 23947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:56.136746 23947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:08:56.138127 23947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:43673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:56.138422 23947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:56.138738 23947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:56.144122 23952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:56.144140 23947 server_base.cc:1034] running on GCE node
W20260925 10:08:56.144091 23953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:56.144110 23955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:56.144659 23947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:56.144986 23947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:56.146152 23947 hybrid_clock.cc:648] HybridClock initialized: now 1790330936146139 us; error 27 us; skew 500 ppm
I20260925 10:08:56.147588 23947 webserver.cc:460] Webserver started at http://127.14.41.66:42321/ using document root <none> and password file <none>
I20260925 10:08:56.147915 23947 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:56.148025 23947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:56.148205 23947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:56.149335 23947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "f514ecc4733b433289de0882336d7495"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "212c2f83a086788ed1397fa488aea860"
server_key_iv: "66bb27a649452b2478960b17c8d3d30f"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.149763 23947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "f514ecc4733b433289de0882336d7495"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "212c2f83a086788ed1397fa488aea860"
server_key_iv: "66bb27a649452b2478960b17c8d3d30f"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.151160 23947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:08:56.151993 23961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:56.152173 23947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:08:56.152277 23947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "f514ecc4733b433289de0882336d7495"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "212c2f83a086788ed1397fa488aea860"
server_key_iv: "66bb27a649452b2478960b17c8d3d30f"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.152385 23947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:56.185140 23947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:56.185513 23947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:56.185681 23947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:56.185941 23947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:56.186326 23947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:56.186386 23947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:56.186421 23947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:56.186448 23947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:56.192667 23947 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:43727
I20260925 10:08:56.192705 24074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:43727 every 8 connection(s)
I20260925 10:08:56.193099 23947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:08:56.195536 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 23947
I20260925 10:08:56.195609 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:08:56.195817 14501 external_mini_cluster.cc:1428] Setting key 0b0605a98aac52a4fb13558ea284824a
I20260925 10:08:56.196807 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37469
--builtin_ntp_servers=127.14.41.84:43673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 10:08:56.199095 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:56.194005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:44729 (local address 127.14.41.126:37469)
0925 10:08:56.194076 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:56.194078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:56.194302 (+ 224us) server_negotiation.cc:407] Connection header received
0925 10:08:56.194845 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:56.194846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:56.194858 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:56.194934 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:56.195715 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:56.196327 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:56.197441 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:56.197647 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:56.197739 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:56.198706 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:56.198707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:56.198708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:56.198712 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:56.198727 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:56.198731 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:56.198771 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:56.198924 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 10:08:56.198951 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:08:56.199352 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:56.193977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:37469 (local address 127.14.41.66:44729)
0925 10:08:56.194161 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:56.194206 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:56.194718 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:56.195017 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:56.195020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:56.195029 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:56.195558 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:56.195562 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:56.196432 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:56.196434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:56.197332 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:56.197334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:56.197751 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:56.197752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:56.197927 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:56.198522 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:08:56.198522 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:56.198568 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:08:56.198600 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:56.198796 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:56.198802 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:56.198805 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:56.198894 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 10:08:56.199030 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260925 10:08:56.199958 24075 heartbeater.cc:347] Connected to a master server at 127.14.41.126:37469
I20260925 10:08:56.200068 24075 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:56.200260 24075 heartbeater.cc:511] Master 127.14.41.126:37469 requested a full tablet report, sending...
I20260925 10:08:56.200676 23756 ts_manager.cc:194] Registered new tserver with Master: f514ecc4733b433289de0882336d7495 (127.14.41.66:43727)
I20260925 10:08:56.201064 23756 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.41.66:44729
W20260925 10:08:56.281304 24078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:56.281618 24078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:56.281648 24078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:56.282884 24078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:56.282934 24078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:56.282960 24078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:08:56.284371 24078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:43673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:56.284646 24078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:56.284966 24078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:56.290470 24084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:56.290612 24078 server_base.cc:1034] running on GCE node
W20260925 10:08:56.290470 24083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:56.290498 24086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:56.291003 24078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:56.291296 24078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:56.292457 24078 hybrid_clock.cc:648] HybridClock initialized: now 1790330936292436 us; error 37 us; skew 500 ppm
I20260925 10:08:56.294121 24078 webserver.cc:460] Webserver started at http://127.14.41.67:34463/ using document root <none> and password file <none>
I20260925 10:08:56.294397 24078 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:56.294453 24078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:56.294659 24078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:56.295991 24078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "015124ad82f446e9b9653cc3045bbd7e"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "203c19f64074ff3d1d22fb93c9e369ce"
server_key_iv: "a726b425708c5e1ec5c649aba07f1c68"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.296423 24078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "015124ad82f446e9b9653cc3045bbd7e"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "203c19f64074ff3d1d22fb93c9e369ce"
server_key_iv: "a726b425708c5e1ec5c649aba07f1c68"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.297865 24078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:08:56.298669 24092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:56.298897 24078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:56.299026 24078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "015124ad82f446e9b9653cc3045bbd7e"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "203c19f64074ff3d1d22fb93c9e369ce"
server_key_iv: "a726b425708c5e1ec5c649aba07f1c68"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.299115 24078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:56.319293 24078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:56.319660 24078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:56.319912 24078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:56.320247 24078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:56.320638 24078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:56.320712 24078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:56.320751 24078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:56.320785 24078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:56.327112 24078 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:36753
I20260925 10:08:56.327155 24205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:36753 every 8 connection(s)
I20260925 10:08:56.327565 24078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:08:56.332718 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 24078
I20260925 10:08:56.332798 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:08:56.333041 14501 external_mini_cluster.cc:1428] Setting key 0a1633dc6a5ed5173708d1b9e3c943e4
I20260925 10:08:56.333209 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:56.328476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:45841 (local address 127.14.41.126:37469)
0925 10:08:56.328555 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:56.328557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:56.328816 (+ 259us) server_negotiation.cc:407] Connection header received
0925 10:08:56.329308 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:56.329309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:56.329320 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:56.329373 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:56.330055 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:56.330592 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:56.331591 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:56.331786 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:56.331905 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:56.332906 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:56.332907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:56.332907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:56.332913 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:56.332933 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:56.332938 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:56.333003 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:56.333095 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 10:08:56.333129 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:08:56.333465 24208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:56.328465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:37469 (local address 127.14.41.67:45841)
0925 10:08:56.328685 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:56.328720 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:56.329219 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:56.329443 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:56.329447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:56.329455 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:56.329968 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:56.329972 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:56.330678 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:56.330679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:56.331509 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:56.331512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:56.331974 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:56.331975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:56.332104 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:56.332692 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:08:56.332692 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:56.332722 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:08:56.332746 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:56.333011 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:56.333016 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:56.333019 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:56.333077 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 10:08:56.333200 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":116,"threads_started":1}
I20260925 10:08:56.333823 24206 heartbeater.cc:347] Connected to a master server at 127.14.41.126:37469
I20260925 10:08:56.333974 24206 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:56.334164 24206 heartbeater.cc:511] Master 127.14.41.126:37469 requested a full tablet report, sending...
I20260925 10:08:56.334519 23756 ts_manager.cc:194] Registered new tserver with Master: 015124ad82f446e9b9653cc3045bbd7e (127.14.41.67:36753)
I20260925 10:08:56.334923 23756 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.41.67:45841
I20260925 10:08:56.344945 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 10:08:56.346992 23799 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:56.346445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.14.41.126:37469)
0925 10:08:56.346509 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:56.346511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:56.346612 (+ 101us) server_negotiation.cc:407] Connection header received
0925 10:08:56.346659 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:56.346660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:56.346673 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:56.346728 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:56.346885 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50866: BlockingRecv error: recv got EOF from 127.0.0.1:50866 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 10:08:56.347050 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 23816
I20260925 10:08:56.351887 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 23947
I20260925 10:08:56.356671 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 24078
I20260925 10:08:56.361081 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 23725
2026-09-25T10:08:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (662 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T10:08:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:56Z Disabled control of system clock
I20260925 10:08:56.389572 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:45043
--webserver_interface=127.14.41.126
--webserver_port=0
--builtin_ntp_servers=127.14.41.84:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:45043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 10:08:56.472165 24217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:56.472402 24217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:56.472425 24217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:56.473589 24217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:56.473634 24217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:56.473647 24217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:56.473660 24217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:56.473672 24217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:08:56.475054 24217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:45043
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:45043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:56.475287 24217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:56.475549 24217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:08:56.480835 24217 server_base.cc:1034] running on GCE node
W20260925 10:08:56.480712 24225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:56.480706 24223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:56.480723 24222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:56.481256 24217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:56.481606 24217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:56.482777 24217 hybrid_clock.cc:648] HybridClock initialized: now 1790330936482754 us; error 60 us; skew 500 ppm
I20260925 10:08:56.484218 24217 webserver.cc:460] Webserver started at http://127.14.41.126:45565/ using document root <none> and password file <none>
I20260925 10:08:56.484526 24217 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:56.484606 24217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:56.484764 24217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:56.485904 24217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "9f5d5470d10d4c3895f8edbd477bc80d"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "bf5339641df50e25918c2c1d26032854"
server_key_iv: "2d1bfd6170cd61b7f23b6239c3facb42"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.486296 24217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "9f5d5470d10d4c3895f8edbd477bc80d"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "bf5339641df50e25918c2c1d26032854"
server_key_iv: "2d1bfd6170cd61b7f23b6239c3facb42"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.487673 24217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:08:56.488588 24231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:56.488801 24217 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:08:56.488912 24217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "9f5d5470d10d4c3895f8edbd477bc80d"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "bf5339641df50e25918c2c1d26032854"
server_key_iv: "2d1bfd6170cd61b7f23b6239c3facb42"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.489001 24217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:56.493116 24217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:56.493404 24217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:56.493558 24217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:56.497555 24217 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:45043
I20260925 10:08:56.497591 24283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:45043 every 8 connection(s)
I20260925 10:08:56.497943 24217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:56.498646 24284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:56.501664 24284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d: Bootstrap starting.
I20260925 10:08:56.502297 24284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:56.502624 24284 log.cc:824] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:56.503388 24284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d: No bootstrap required, opened a new log
I20260925 10:08:56.504513 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 24217
I20260925 10:08:56.504621 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:08:56.504845 14501 external_mini_cluster.cc:1428] Setting key 9579134e37df240fbba606370c29027e
I20260925 10:08:56.505777 24284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5d5470d10d4c3895f8edbd477bc80d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45043 } }
I20260925 10:08:56.505928 24284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:56.505951 24284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f5d5470d10d4c3895f8edbd477bc80d, State: Initialized, Role: FOLLOWER
I20260925 10:08:56.506146 24284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5d5470d10d4c3895f8edbd477bc80d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45043 } }
I20260925 10:08:56.506249 24284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:08:56.506291 24284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:56.506341 24284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:56.507157 24284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5d5470d10d4c3895f8edbd477bc80d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45043 } }
I20260925 10:08:56.507239 24288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:56.505445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55066 (local address 127.14.41.126:45043)
0925 10:08:56.505670 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:56.505679 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:56.505705 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:08:56.505808 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:56.505813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:56.505850 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:56.506017 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:56.506746 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:56.506750 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:56.506750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:56.506753 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:56.506800 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:56.506811 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:56.506874 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:56.506970 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 10:08:56.507055 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20260925 10:08:56.507282 24284 leader_election.cc:302] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f5d5470d10d4c3895f8edbd477bc80d; no voters:
I20260925 10:08:56.507476 24284 leader_election.cc:288] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:08:56.507753 24289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:56.507970 24289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d [term 1 LEADER]: Becoming Leader. State: Replica: 9f5d5470d10d4c3895f8edbd477bc80d, State: Running, Role: LEADER
I20260925 10:08:56.508070 24289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5d5470d10d4c3895f8edbd477bc80d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45043 } }
I20260925 10:08:56.508193 24284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:56.508523 24290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f5d5470d10d4c3895f8edbd477bc80d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5d5470d10d4c3895f8edbd477bc80d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45043 } } }
I20260925 10:08:56.508610 24290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:56.508873 24291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f5d5470d10d4c3895f8edbd477bc80d. Latest consensus state: current_term: 1 leader_uuid: "9f5d5470d10d4c3895f8edbd477bc80d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f5d5470d10d4c3895f8edbd477bc80d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45043 } } }
I20260925 10:08:56.509002 24291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:56.509115 24293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:56.509738 24293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:56.511432 24293 catalog_manager.cc:1347] Generated new cluster ID: cbd1351f69714cfeac54e04376052d30
I20260925 10:08:56.511492 24293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:08:56.519455 24293 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:08:56.519670 24293 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:56.529124 24293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f5d5470d10d4c3895f8edbd477bc80d: Generated new TSK 0
I20260925 10:08:56.529381 24293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:08:56.532810 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:45043
--builtin_ntp_servers=127.14.41.84:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 10:08:56.624732 24308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:56.624991 24308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:56.625032 24308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:56.626250 24308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:56.626330 24308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:56.626377 24308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:08:56.627781 24308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:45043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:56.628095 24308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:56.628419 24308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:56.633735 24313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:56.633859 24316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:56.633987 24308 server_base.cc:1034] running on GCE node
W20260925 10:08:56.634042 24314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:56.634328 24308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:56.634613 24308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:56.635807 24308 hybrid_clock.cc:648] HybridClock initialized: now 1790330936635778 us; error 47 us; skew 500 ppm
I20260925 10:08:56.637377 24308 webserver.cc:460] Webserver started at http://127.14.41.65:43763/ using document root <none> and password file <none>
I20260925 10:08:56.637652 24308 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:56.637727 24308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:56.637887 24308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:56.639029 24308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "a7cfcfc78ae6403ea023f015da01c545"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "98a4175d477f8060fa3192ed86979c5d"
server_key_iv: "0b9fab20f84621b304c9e1d3e284f754"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.639464 24308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "a7cfcfc78ae6403ea023f015da01c545"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "98a4175d477f8060fa3192ed86979c5d"
server_key_iv: "0b9fab20f84621b304c9e1d3e284f754"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.641026 24308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:08:56.641889 24322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:56.642091 24308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:56.642194 24308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "a7cfcfc78ae6403ea023f015da01c545"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "98a4175d477f8060fa3192ed86979c5d"
server_key_iv: "0b9fab20f84621b304c9e1d3e284f754"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.642289 24308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:56.646579 24308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:56.646891 24308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:56.647033 24308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:56.647269 24308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:56.647656 24308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:56.647715 24308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:56.647760 24308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:56.647789 24308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:56.654860 24308 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:34011
I20260925 10:08:56.654902 24435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:34011 every 8 connection(s)
I20260925 10:08:56.655259 24308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:08:56.658282 24288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:56.656102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:41141 (local address 127.14.41.126:45043)
0925 10:08:56.656149 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:56.656151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:56.656394 (+ 243us) server_negotiation.cc:407] Connection header received
0925 10:08:56.656880 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:56.656880 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:56.656888 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:56.656944 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:56.658089 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:56.658091 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:56.658091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:56.658095 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:56.658112 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:56.658115 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:56.658165 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:56.658214 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 10:08:56.658228 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:08:56.658545 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:56.656083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:45043 (local address 127.14.41.65:41141)
0925 10:08:56.656262 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:56.656299 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:56.656793 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:56.657002 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:56.657005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:56.657014 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:56.657925 (+ 911us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:08:56.657925 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:56.657956 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:08:56.657988 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:56.658163 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:56.658165 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:56.658169 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:08:56.658222 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 10:08:56.658274 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260925 10:08:56.658865 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 24308
I20260925 10:08:56.658968 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:08:56.659092 24436 heartbeater.cc:347] Connected to a master server at 127.14.41.126:45043
I20260925 10:08:56.659204 14501 external_mini_cluster.cc:1428] Setting key b28e3d776d55aa4ad01bb8c7acbdb677
I20260925 10:08:56.659209 24436 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:56.659430 24436 heartbeater.cc:511] Master 127.14.41.126:45043 requested a full tablet report, sending...
I20260925 10:08:56.660099 24247 ts_manager.cc:194] Registered new tserver with Master: a7cfcfc78ae6403ea023f015da01c545 (127.14.41.65:34011)
I20260925 10:08:56.660328 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:45043
--builtin_ntp_servers=127.14.41.84:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 10:08:56.747752 24439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:56.748064 24439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:56.748095 24439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:56.749286 24439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:56.749325 24439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:56.749351 24439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:08:56.750665 24439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:45043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:56.751132 24439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:56.751415 24439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:56.756803 24444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:56.756847 24445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:56.757030 24439 server_base.cc:1034] running on GCE node
W20260925 10:08:56.756891 24447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:56.757329 24439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:56.757633 24439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:56.758852 24439 hybrid_clock.cc:648] HybridClock initialized: now 1790330936758821 us; error 38 us; skew 500 ppm
I20260925 10:08:56.760354 24439 webserver.cc:460] Webserver started at http://127.14.41.66:44723/ using document root <none> and password file <none>
I20260925 10:08:56.760632 24439 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:56.760825 24439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:56.760984 24439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:56.762118 24439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "c3635f97c6eb4fd1b60eb537826a9a5f"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "3eb907c97be6210c875c47b29a375d58"
server_key_iv: "cd2ebbaff0368a850bdd8cf49fab2a36"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.762495 24439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "c3635f97c6eb4fd1b60eb537826a9a5f"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "3eb907c97be6210c875c47b29a375d58"
server_key_iv: "cd2ebbaff0368a850bdd8cf49fab2a36"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.764029 24439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:08:56.764948 24453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:56.765208 24439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:56.765326 24439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "c3635f97c6eb4fd1b60eb537826a9a5f"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "3eb907c97be6210c875c47b29a375d58"
server_key_iv: "cd2ebbaff0368a850bdd8cf49fab2a36"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.765412 24439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:56.769923 24439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:56.770262 24439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:56.770418 24439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:56.770669 24439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:56.771021 24439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:56.771059 24439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:56.771128 24439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:56.771158 24439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:56.777634 24439 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:40353
I20260925 10:08:56.777707 24566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:40353 every 8 connection(s)
I20260925 10:08:56.778111 24439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:08:56.781749 24288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:56.779133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:48861 (local address 127.14.41.126:45043)
0925 10:08:56.779201 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:56.779203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:56.779487 (+ 284us) server_negotiation.cc:407] Connection header received
0925 10:08:56.780073 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:56.780074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:56.780081 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:56.780149 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:56.781398 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:56.781399 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:56.781399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:56.781403 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:56.781420 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:56.781425 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:56.781534 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:56.781595 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 10:08:56.781608 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 10:08:56.781896 24569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:56.779139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:45043 (local address 127.14.41.66:48861)
0925 10:08:56.779352 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:56.779394 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:56.779865 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:56.780226 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:56.780230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:56.780239 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:56.781228 (+ 989us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:08:56.781228 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:56.781258 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:08:56.781288 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:56.781530 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:56.781532 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:56.781534 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:08:56.781586 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 10:08:56.781634 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20260925 10:08:56.782389 24567 heartbeater.cc:347] Connected to a master server at 127.14.41.126:45043
I20260925 10:08:56.782492 24567 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:56.782665 24567 heartbeater.cc:511] Master 127.14.41.126:45043 requested a full tablet report, sending...
I20260925 10:08:56.783051 24247 ts_manager.cc:194] Registered new tserver with Master: c3635f97c6eb4fd1b60eb537826a9a5f (127.14.41.66:40353)
I20260925 10:08:56.785562 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 24439
I20260925 10:08:56.785646 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:08:56.785881 14501 external_mini_cluster.cc:1428] Setting key 14932de351cc0b26ad766d98b01d7772
I20260925 10:08:56.787098 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:45043
--builtin_ntp_servers=127.14.41.84:46671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 10:08:56.870469 24570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:56.870689 24570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:56.870714 24570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:56.871999 24570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:56.872043 24570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:56.872071 24570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:08:56.873508 24570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:46671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:45043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:56.873780 24570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:56.874085 24570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:56.879479 24575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:56.879465 24576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:56.879467 24578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:56.879739 24570 server_base.cc:1034] running on GCE node
I20260925 10:08:56.880079 24570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:56.880342 24570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:56.881553 24570 hybrid_clock.cc:648] HybridClock initialized: now 1790330936881530 us; error 38 us; skew 500 ppm
I20260925 10:08:56.883050 24570 webserver.cc:460] Webserver started at http://127.14.41.67:34765/ using document root <none> and password file <none>
I20260925 10:08:56.883415 24570 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:56.883504 24570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:56.883682 24570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:56.885059 24570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "8396004ff1214915942bb4a6e0a39939"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "06fcb01fccaf71c04330393af2f32b06"
server_key_iv: "c3767f0bb31e736f206deda8d86ca775"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.885438 24570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "8396004ff1214915942bb4a6e0a39939"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "06fcb01fccaf71c04330393af2f32b06"
server_key_iv: "c3767f0bb31e736f206deda8d86ca775"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.886871 24570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:08:56.887820 24584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:56.888046 24570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:56.888190 24570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "8396004ff1214915942bb4a6e0a39939"
format_stamp: "Formatted at 2026-09-25 10:08:56 on dist-test-slave-s6tm"
server_key: "06fcb01fccaf71c04330393af2f32b06"
server_key_iv: "c3767f0bb31e736f206deda8d86ca775"
server_key_version: "encryptionkey@0"
I20260925 10:08:56.888276 24570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:56.891709 24570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:56.892078 24570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:56.892230 24570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:56.892432 24570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:56.892793 24570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:56.892851 24570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:56.892892 24570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:56.892920 24570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:56.898919 24570 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:44643
I20260925 10:08:56.898959 24697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:44643 every 8 connection(s)
I20260925 10:08:56.899315 24570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:08:56.902369 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 24570
I20260925 10:08:56.902473 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:08:56.902565 24288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:56.900155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:47851 (local address 127.14.41.126:45043)
0925 10:08:56.900239 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:56.900242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:56.900486 (+ 244us) server_negotiation.cc:407] Connection header received
0925 10:08:56.900970 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:56.900971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:56.900977 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:56.901042 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:56.902309 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:56.902310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:56.902310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:08:56.902315 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:56.902333 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:08:56.902338 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:56.902414 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:56.902468 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 10:08:56.902485 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:08:56.902693 14501 external_mini_cluster.cc:1428] Setting key 2cd69a35e6855bea691a1310d8d9012c
I20260925 10:08:56.902726 24700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:56.900134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:45043 (local address 127.14.41.67:47851)
0925 10:08:56.900366 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:56.900402 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:56.900855 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:56.901098 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:56.901101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:56.901109 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:56.902132 (+ 1023us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:08:56.902132 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:56.902162 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:08:56.902193 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:56.902410 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:56.902412 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:56.902414 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:08:56.902466 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 10:08:56.902513 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":106,"threads_started":1}
I20260925 10:08:56.903366 24698 heartbeater.cc:347] Connected to a master server at 127.14.41.126:45043
I20260925 10:08:56.903478 24698 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:56.903626 24698 heartbeater.cc:511] Master 127.14.41.126:45043 requested a full tablet report, sending...
I20260925 10:08:56.904053 24247 ts_manager.cc:194] Registered new tserver with Master: 8396004ff1214915942bb4a6e0a39939 (127.14.41.67:44643)
I20260925 10:08:56.914608 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 10:08:56.916791 24288 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:08:56.916069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55070 (local address 127.14.41.126:45043)
0925 10:08:56.916153 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:56.916156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:56.916302 (+ 146us) server_negotiation.cc:407] Connection header received
0925 10:08:56.916386 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:56.916387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:56.916393 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:56.916463 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:56.916618 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55070: BlockingRecv error: recv got EOF from 127.0.0.1:55070 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:08:56.916786 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 24308
I20260925 10:08:56.921363 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 24439
I20260925 10:08:56.925822 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 24570
I20260925 10:08:56.930677 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 24217
2026-09-25T10:08:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (569 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:08:56 dist-test-slave-s6tm krb5kdc[24708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:08:56 dist-test-slave-s6tm krb5kdc[24708](info): set up 2 sockets
Sep 25 10:08:56 dist-test-slave-s6tm krb5kdc[24708](info): commencing operation
krb5kdc: starting...
W20260925 10:08:57.982425 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:08:58 dist-test-slave-s6tm krb5kdc[24708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:08:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:08:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:08:58.108047 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:46521
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:46521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:58.193297 24721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:58.193579 24721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:58.193621 24721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:58.194901 24721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:08:58.194986 24721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:58.195011 24721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:58.195044 24721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:08:58.195070 24721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:08:58.196786 24721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:46521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:46521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:58.197093 24721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:58.197388 24721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:58.203059 24729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:58.203052 24727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:58.203277 24726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:58.203325 24721 server_base.cc:1034] running on GCE node
I20260925 10:08:58.203758 24721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:58.204094 24721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:58.205247 24721 hybrid_clock.cc:648] HybridClock initialized: now 1790330938205206 us; error 52 us; skew 500 ppm
Sep 25 10:08:58 dist-test-slave-s6tm krb5kdc[24708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330938, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:58.208254 24721 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:08:58.208868 24721 webserver.cc:460] Webserver started at http://127.14.41.126:43323/ using document root <none> and password file <none>
I20260925 10:08:58.209136 24721 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:58.209213 24721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:58.209367 24721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:58.210512 24721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "16d27b71e6114e32bdbace34439f2733"
format_stamp: "Formatted at 2026-09-25 10:08:58 on dist-test-slave-s6tm"
server_key: "c14f1d8051dabbd3a4ee189ce235a0d7"
server_key_iv: "aa5d18c0273a6a0a482ca233e0497965"
server_key_version: "encryptionkey@0"
I20260925 10:08:58.210952 24721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "16d27b71e6114e32bdbace34439f2733"
format_stamp: "Formatted at 2026-09-25 10:08:58 on dist-test-slave-s6tm"
server_key: "c14f1d8051dabbd3a4ee189ce235a0d7"
server_key_iv: "aa5d18c0273a6a0a482ca233e0497965"
server_key_version: "encryptionkey@0"
I20260925 10:08:58.212543 24721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:08:58.213642 24736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:58.213943 24721 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:58.214059 24721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "16d27b71e6114e32bdbace34439f2733"
format_stamp: "Formatted at 2026-09-25 10:08:58 on dist-test-slave-s6tm"
server_key: "c14f1d8051dabbd3a4ee189ce235a0d7"
server_key_iv: "aa5d18c0273a6a0a482ca233e0497965"
server_key_version: "encryptionkey@0"
I20260925 10:08:58.214155 24721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:58.236474 24721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:58.238687 24721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:58.238967 24721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:58.243216 24721 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:46521
I20260925 10:08:58.243252 24788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:46521 every 8 connection(s)
I20260925 10:08:58.243626 24721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:08:58.244005 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 24721
I20260925 10:08:58.244115 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:08:58.244354 14501 external_mini_cluster.cc:1428] Setting key eb6537aa7bf091f98ec432b6c81f8afd
I20260925 10:08:58.244781 24789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:58.247761 24789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733: Bootstrap starting.
I20260925 10:08:58.248471 24789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:58.248790 24789 log.cc:824] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:58.249492 24789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733: No bootstrap required, opened a new log
Sep 25 10:08:58 dist-test-slave-s6tm krb5kdc[24708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:58.251618 24789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d27b71e6114e32bdbace34439f2733" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46521 } }
I20260925 10:08:58.251757 24789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:58.251780 24789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d27b71e6114e32bdbace34439f2733, State: Initialized, Role: FOLLOWER
I20260925 10:08:58.251955 24789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d27b71e6114e32bdbace34439f2733" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46521 } }
I20260925 10:08:58.252066 24789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:08:58.252111 24789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:08:58.252168 24789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:58.253008 24789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d27b71e6114e32bdbace34439f2733" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46521 } }
I20260925 10:08:58.253134 24789 leader_election.cc:302] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16d27b71e6114e32bdbace34439f2733; no voters:
I20260925 10:08:58.253321 24789 leader_election.cc:288] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:08:58.253402 24794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:58.253548 24794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733 [term 1 LEADER]: Becoming Leader. State: Replica: 16d27b71e6114e32bdbace34439f2733, State: Running, Role: LEADER
I20260925 10:08:58.253587 24789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:08:58.254365 24794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d27b71e6114e32bdbace34439f2733" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46521 } }
I20260925 10:08:58.255146 24795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16d27b71e6114e32bdbace34439f2733. Latest consensus state: current_term: 1 leader_uuid: "16d27b71e6114e32bdbace34439f2733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d27b71e6114e32bdbace34439f2733" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46521 } } }
I20260925 10:08:58.255245 24795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:58.255429 24796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16d27b71e6114e32bdbace34439f2733" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d27b71e6114e32bdbace34439f2733" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46521 } } }
I20260925 10:08:58.255506 24796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733 [sys.catalog]: This master's current role is: LEADER
I20260925 10:08:58.256052 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:58.244949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57486 (local address 127.14.41.126:46521)
0925 10:08:58.245149 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:58.245159 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:58.245192 (+ 33us) server_negotiation.cc:407] Connection header received
0925 10:08:58.245945 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:58.245953 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:58.246009 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:58.246175 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:58.246831 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.247726 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.248568 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.249052 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.249180 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:58.251895 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:58.251899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:58.251900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:58.251912 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:58.253934 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:58.254411 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:58.254414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:58.254415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:58.254530 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:58.254943 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:58.254944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:58.254944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:58.255306 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:58.255464 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:58.255706 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 10:08:58.255837 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":85,"threads_started":1}
W20260925 10:08:58.256440 24808 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:08:58.256531 24808 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:08:58.256757 24809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:08:58.256969 24809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:08:58.258594 24809 catalog_manager.cc:1347] Generated new cluster ID: 01e4123ad3e949aa840e73dceff05e28
I20260925 10:08:58.258674 24809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:08:58.265208 24809 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:08:58.266005 24809 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:08:58.276989 24809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733: Generated new TSK 0
I20260925 10:08:58.277244 24809 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
I20260925 10:08:58.315658 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46521
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:58.418642 24815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:58.418859 24815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:58.418884 24815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:58.420133 24815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:58.420182 24815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:58.420210 24815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:08:58.421835 24815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:58.422097 24815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:58.422358 24815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:58.427816 24823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:58.427973 24820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:58.427981 24821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:58.428097 24815 server_base.cc:1034] running on GCE node
I20260925 10:08:58.428372 24815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:58.428606 24815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:58.429754 24815 hybrid_clock.cc:648] HybridClock initialized: now 1790330938429730 us; error 39 us; skew 500 ppm
Sep 25 10:08:58 dist-test-slave-s6tm krb5kdc[24708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330938, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:58.432487 24815 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:08:58.432925 24815 webserver.cc:460] Webserver started at http://127.14.41.65:43237/ using document root <none> and password file <none>
I20260925 10:08:58.433120 24815 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:58.433161 24815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:58.433255 24815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:58.434319 24815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "4a3641f4f5514331a4b6738b86beee72"
format_stamp: "Formatted at 2026-09-25 10:08:58 on dist-test-slave-s6tm"
server_key: "9edd5801afbf49c11ae336844a61a7b4"
server_key_iv: "edc669cbff0cc8b5cf22ee3d947948c8"
server_key_version: "encryptionkey@0"
I20260925 10:08:58.434674 24815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "4a3641f4f5514331a4b6738b86beee72"
format_stamp: "Formatted at 2026-09-25 10:08:58 on dist-test-slave-s6tm"
server_key: "9edd5801afbf49c11ae336844a61a7b4"
server_key_iv: "edc669cbff0cc8b5cf22ee3d947948c8"
server_key_version: "encryptionkey@0"
I20260925 10:08:58.436071 24815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:08:58.436913 24830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:58.437119 24815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:58.437252 24815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "4a3641f4f5514331a4b6738b86beee72"
format_stamp: "Formatted at 2026-09-25 10:08:58 on dist-test-slave-s6tm"
server_key: "9edd5801afbf49c11ae336844a61a7b4"
server_key_iv: "edc669cbff0cc8b5cf22ee3d947948c8"
server_key_version: "encryptionkey@0"
I20260925 10:08:58.437312 24815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:58.462257 24815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:58.464787 24815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:58.464906 24815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:58.465227 24815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:58.465564 24815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:58.465600 24815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:58.465622 24815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:58.465668 24815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:58.471582 24815 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:34315
I20260925 10:08:58.471642 24943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:34315 every 8 connection(s)
I20260925 10:08:58.472016 24815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
Sep 25 10:08:58 dist-test-slave-s6tm krb5kdc[24708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330938, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:58.482065 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 24815
I20260925 10:08:58.482254 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:08:58.482465 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:58.472857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:37033 (local address 127.14.41.126:46521)
0925 10:08:58.472958 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:58.472961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:58.473169 (+ 208us) server_negotiation.cc:407] Connection header received
0925 10:08:58.474204 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:58.474205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:58.474218 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:58.474270 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:58.475036 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.475563 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.476693 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.476919 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.477014 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:58.479354 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:58.479356 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:58.479356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:58.479369 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:58.480796 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:58.481268 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:58.481270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:58.481270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:58.481332 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:58.481610 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:58.481611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:58.481611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:58.481757 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:58.481845 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:58.482231 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 10:08:58.482379 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 10:08:58.482476 14501 external_mini_cluster.cc:1428] Setting key b4f7722b859563eb30c91cae604b8d9e
I20260925 10:08:58.482594 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:58.472813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46521 (local address 127.14.41.65:37033)
0925 10:08:58.473010 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:58.473063 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:58.474066 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:58.474365 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:58.474370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:58.474653 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:58.474907 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:58.474912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.475675 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.475676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:58.476591 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:58.476595 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.477026 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.477027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:58.477163 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:58.477777 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:58.477785 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:58.479239 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:58.480919 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:58.480928 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:58.480930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:58.481184 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:58.481415 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:58.481416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:58.481416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:58.481529 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:58.481835 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:58.481843 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:58.482022 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 10:08:58.482151 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 10:08:58.482344 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":88,"threads_started":1}
I20260925 10:08:58.483104 24944 heartbeater.cc:347] Connected to a master server at 127.14.41.126:46521
I20260925 10:08:58.483232 24944 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:58.483453 24944 heartbeater.cc:511] Master 127.14.41.126:46521 requested a full tablet report, sending...
I20260925 10:08:58.484088 24752 ts_manager.cc:194] Registered new tserver with Master: 4a3641f4f5514331a4b6738b86beee72 (127.14.41.65:34315)
I20260925 10:08:58.485015 24752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:37033
WARNING: no policy specified for kudu/127.14.41.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
I20260925 10:08:58.512035 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46521
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:58.596729 24949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:58.596988 24949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:58.597030 24949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:58.598268 24949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:58.598347 24949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:58.598392 24949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:08:58.600013 24949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:58.600328 24949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:58.600580 24949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:58.606048 24955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:58.606007 24954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:58.606007 24957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:58.606524 24949 server_base.cc:1034] running on GCE node
I20260925 10:08:58.606788 24949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:58.607092 24949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:58.608269 24949 hybrid_clock.cc:648] HybridClock initialized: now 1790330938608239 us; error 41 us; skew 500 ppm
Sep 25 10:08:58 dist-test-slave-s6tm krb5kdc[24708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330938, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:58.611177 24949 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:08:58.611661 24949 webserver.cc:460] Webserver started at http://127.14.41.66:42669/ using document root <none> and password file <none>
I20260925 10:08:58.611984 24949 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:58.612066 24949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:58.612233 24949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:58.613358 24949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "9894174acee14ea6953d8bec3d3dc193"
format_stamp: "Formatted at 2026-09-25 10:08:58 on dist-test-slave-s6tm"
server_key: "8cd09505142b268b717b532a076c0f4e"
server_key_iv: "2e22bd07c90d0c0a5b198e2929fd38ad"
server_key_version: "encryptionkey@0"
I20260925 10:08:58.613770 24949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "9894174acee14ea6953d8bec3d3dc193"
format_stamp: "Formatted at 2026-09-25 10:08:58 on dist-test-slave-s6tm"
server_key: "8cd09505142b268b717b532a076c0f4e"
server_key_iv: "2e22bd07c90d0c0a5b198e2929fd38ad"
server_key_version: "encryptionkey@0"
I20260925 10:08:58.615149 24949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:08:58.616120 24964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:58.616303 24949 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:08:58.616410 24949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "9894174acee14ea6953d8bec3d3dc193"
format_stamp: "Formatted at 2026-09-25 10:08:58 on dist-test-slave-s6tm"
server_key: "8cd09505142b268b717b532a076c0f4e"
server_key_iv: "2e22bd07c90d0c0a5b198e2929fd38ad"
server_key_version: "encryptionkey@0"
I20260925 10:08:58.616488 24949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:58.628899 24949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:58.631047 24949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:58.631251 24949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:58.631546 24949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:58.632025 24949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:58.632081 24949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:58.632133 24949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:58.632179 24949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:58.638489 24949 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:45141
I20260925 10:08:58.638540 25077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:45141 every 8 connection(s)
I20260925 10:08:58.638898 24949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:08:58 dist-test-slave-s6tm krb5kdc[24708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330938, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:58.647730 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 24949
I20260925 10:08:58.647861 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:08:58.648108 14501 external_mini_cluster.cc:1428] Setting key a6fabf2f3e010ca15b5179002d462564
I20260925 10:08:58.649731 25080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:58.639635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46521 (local address 127.14.41.66:37133)
0925 10:08:58.639798 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:58.639842 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:58.640410 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:58.640926 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:58.640930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:58.641549 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:58.641817 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:58.641823 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.642552 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.642553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:58.643404 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:58.643406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.643793 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.643794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:58.643918 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:58.644564 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:58.644573 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:58.646041 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:58.647996 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:58.648009 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:58.648010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:58.648314 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:58.648617 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:58.648618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:58.648618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:58.648762 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:58.649153 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:58.649158 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:58.649288 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 10:08:58.649401 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 10:08:58.649554 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260925 10:08:58.649824 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:58.639662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:37133 (local address 127.14.41.126:46521)
0925 10:08:58.639711 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:58.639713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:58.640099 (+ 386us) server_negotiation.cc:407] Connection header received
0925 10:08:58.640526 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:58.640562 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:58.640577 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:58.640818 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:58.641927 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.642461 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.643493 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.643699 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.643772 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:58.646140 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:58.646141 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:58.646142 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:58.646149 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:58.647859 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:58.648459 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:58.648461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:58.648461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:58.648535 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:58.648872 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:58.648873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:58.648874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:58.649075 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:58.649154 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:58.649467 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 10:08:58.649565 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:08:58.650240 25078 heartbeater.cc:347] Connected to a master server at 127.14.41.126:46521
I20260925 10:08:58.650342 25078 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:58.650540 25078 heartbeater.cc:511] Master 127.14.41.126:46521 requested a full tablet report, sending...
I20260925 10:08:58.650951 24752 ts_manager.cc:194] Registered new tserver with Master: 9894174acee14ea6953d8bec3d3dc193 (127.14.41.66:45141)
I20260925 10:08:58.651335 24752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:37133
WARNING: no policy specified for kudu/127.14.41.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
I20260925 10:08:58.675704 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46521
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:08:58.759863 25083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:08:58.760227 25083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:08:58.760269 25083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:08:58.761505 25083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:08:58.761584 25083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:08:58.761680 25083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:08:58.763468 25083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:08:58.763775 25083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:08:58.764084 25083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:08:58.769697 25089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:08:58.769712 25091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:58.769861 25083 server_base.cc:1034] running on GCE node
W20260925 10:08:58.769984 25088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:08:58.770287 25083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:08:58.770522 25083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:08:58.771690 25083 hybrid_clock.cc:648] HybridClock initialized: now 1790330938771673 us; error 39 us; skew 500 ppm
Sep 25 10:08:58 dist-test-slave-s6tm krb5kdc[24708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330938, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:08:58.774667 25083 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:08:58.775231 25083 webserver.cc:460] Webserver started at http://127.14.41.67:42813/ using document root <none> and password file <none>
I20260925 10:08:58.775455 25083 fs_manager.cc:359] Metadata directory not provided
I20260925 10:08:58.775502 25083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:08:58.775615 25083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:08:58.776902 25083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "6d676fedadd64872b9eaf0d8eabdca26"
format_stamp: "Formatted at 2026-09-25 10:08:58 on dist-test-slave-s6tm"
server_key: "938b8244dde2f52519ab588a54609c0b"
server_key_iv: "72ce9a910e471464208b391755cf7ce7"
server_key_version: "encryptionkey@0"
I20260925 10:08:58.777647 25083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "6d676fedadd64872b9eaf0d8eabdca26"
format_stamp: "Formatted at 2026-09-25 10:08:58 on dist-test-slave-s6tm"
server_key: "938b8244dde2f52519ab588a54609c0b"
server_key_iv: "72ce9a910e471464208b391755cf7ce7"
server_key_version: "encryptionkey@0"
I20260925 10:08:58.779204 25083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:08:58.780174 25098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:58.780364 25083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:08:58.780483 25083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "6d676fedadd64872b9eaf0d8eabdca26"
format_stamp: "Formatted at 2026-09-25 10:08:58 on dist-test-slave-s6tm"
server_key: "938b8244dde2f52519ab588a54609c0b"
server_key_iv: "72ce9a910e471464208b391755cf7ce7"
server_key_version: "encryptionkey@0"
I20260925 10:08:58.780565 25083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:08:58.797248 25083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:08:58.799628 25083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:08:58.799823 25083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:08:58.800112 25083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:08:58.800477 25083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:08:58.800539 25083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:58.800582 25083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:08:58.800606 25083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:08:58.806782 25083 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:35927
I20260925 10:08:58.806833 25211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:35927 every 8 connection(s)
I20260925 10:08:58.807219 25083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:08:58.811301 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 25083
I20260925 10:08:58.811408 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:08:58.811708 14501 external_mini_cluster.cc:1428] Setting key b9a1a86ef7c8df0f338172a07e4ab621
Sep 25 10:08:58 dist-test-slave-s6tm krb5kdc[24708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330938, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:08:58.817842 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:58.808100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:58415 (local address 127.14.41.126:46521)
0925 10:08:58.808147 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:58.808149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:58.808367 (+ 218us) server_negotiation.cc:407] Connection header received
0925 10:08:58.808875 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:58.808877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:58.808888 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:58.808945 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:58.809924 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.810461 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.811902 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.812241 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.812329 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:58.814840 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:58.814842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:58.814842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:58.814851 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:58.816300 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:58.816755 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:58.816756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:58.816756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:58.816817 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:58.817108 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:58.817109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:58.817109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:58.817231 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:58.817305 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:58.817616 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 10:08:58.817736 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:08:58.818001 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:58.808106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46521 (local address 127.14.41.67:58415)
0925 10:08:58.808240 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:58.808278 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:58.808763 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:58.809009 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:58.809013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:58.809605 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:08:58.809809 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:58.809814 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.810572 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.810573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:58.811798 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:58.811801 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.812333 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.812334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:58.812473 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:58.813248 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:08:58.813256 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:08:58.814734 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:08:58.816408 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:58.816416 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:58.816418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:58.816663 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:58.816918 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:08:58.816919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:08:58.816919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:08:58.817030 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:08:58.817306 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:08:58.817314 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:08:58.817476 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 10:08:58.817584 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:08:58.817749 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":68,"threads_started":1}
I20260925 10:08:58.818591 25212 heartbeater.cc:347] Connected to a master server at 127.14.41.126:46521
I20260925 10:08:58.818684 25212 heartbeater.cc:464] Registering TS with master...
I20260925 10:08:58.818934 25212 heartbeater.cc:511] Master 127.14.41.126:46521 requested a full tablet report, sending...
I20260925 10:08:58.819386 24752 ts_manager.cc:194] Registered new tserver with Master: 6d676fedadd64872b9eaf0d8eabdca26 (127.14.41.67:35927)
I20260925 10:08:58.819778 24752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:58415
I20260925 10:08:58.824244 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:08:58.832366 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:58.825678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57490 (local address 127.14.41.126:46521)
0925 10:08:58.825831 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:58.825833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:58.825930 (+ 97us) server_negotiation.cc:407] Connection header received
0925 10:08:58.826012 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:58.826013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:58.826023 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:58.826094 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:08:58.826728 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.827288 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.828121 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.828324 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.828431 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:58.829875 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:08:58.829877 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:08:58.829877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:08:58.829884 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:08:58.831046 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:58.831415 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:58.831416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:58.831417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:58.831478 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:08:58.831694 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:08:58.831695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:08:58.831695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:08:58.831860 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:08:58.832012 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:58.832124 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 10:08:58.832258 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260925 10:08:58.834208 24752 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:08:58.835999 24752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:08:58.842907 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:58.838394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:34315 (local address 127.0.0.1:56524)
0925 10:08:58.838842 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:58.838852 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:58.838919 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:58.839177 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:58.839178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:58.839180 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:58.839215 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:58.839217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.840612 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.840613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:58.841739 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:58.841742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.842546 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.842547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:58.842631 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:58.842646 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 10:08:58.842746 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 10:08:58.842806 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":29,"thread_start_us":78,"threads_started":1}
I20260925 10:08:58.842907 25228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:58.838617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56524 (local address 127.14.41.65:34315)
0925 10:08:58.838787 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:58.838792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:58.838941 (+ 149us) server_negotiation.cc:407] Connection header received
0925 10:08:58.838988 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:58.838990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:58.839007 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:58.839079 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:58.839294 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.840494 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.841831 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.842443 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.842535 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:58.842600 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:58.842766 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 10:08:58.842805 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":70,"threads_started":1}
I20260925 10:08:58.842922 25225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:58.838360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:35927 (local address 127.0.0.1:40158)
0925 10:08:58.838681 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:58.838691 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:58.838732 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:58.839025 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:58.839140 (+ 115us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:58.839143 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:58.839204 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:58.839206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.840234 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.840235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:58.841596 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:58.841600 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.842649 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.842650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:58.842718 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:58.842722 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:08:58.842777 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 10:08:58.842823 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":92,"threads_started":1}
I20260925 10:08:58.843271 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:58.838494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47052 (local address 127.14.41.66:45141)
0925 10:08:58.838840 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:58.838843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:58.838851 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:08:58.838882 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:58.838883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:58.838917 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:58.838993 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:58.839273 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.840485 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.841987 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.842725 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.842784 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:58.842843 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:58.843158 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 10:08:58.843198 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":84,"threads_started":1}
I20260925 10:08:58.843420 25223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:58.838350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:45141 (local address 127.0.0.1:47052)
0925 10:08:58.838522 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:58.838566 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:58.838635 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:58.839089 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:58.839097 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:58.839099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:58.839184 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:58.839186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.840657 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.840658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:58.841898 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:58.841900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.842878 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.842879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:58.842936 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:58.842940 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:08:58.843001 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 10:08:58.843045 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":71,"thread_start_us":66,"threads_started":1}
I20260925 10:08:58.843871 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:58.838598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40158 (local address 127.14.41.67:35927)
0925 10:08:58.838787 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:58.838792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:58.838803 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:08:58.838915 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:58.838918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:58.838936 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:58.839019 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:58.839289 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.840117 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.841693 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:58.842462 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:58.842537 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:58.842599 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:58.842960 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 10:08:58.843793 (+ 833us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
I20260925 10:08:58.843751 24878 tablet_service.cc:1467] Processing CreateTablet for tablet bf14701a02e64de4ad3f58eee3e55fba (DEFAULT_TABLE table=test-table [id=ccbe6a79568f4777acc2eb592185cb25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:58.844214 24878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf14701a02e64de4ad3f58eee3e55fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:58.844266 25010 tablet_service.cc:1467] Processing CreateTablet for tablet bf14701a02e64de4ad3f58eee3e55fba (DEFAULT_TABLE table=test-table [id=ccbe6a79568f4777acc2eb592185cb25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:58.844609 25010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf14701a02e64de4ad3f58eee3e55fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:58.844532 25146 tablet_service.cc:1467] Processing CreateTablet for tablet bf14701a02e64de4ad3f58eee3e55fba (DEFAULT_TABLE table=test-table [id=ccbe6a79568f4777acc2eb592185cb25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:08:58.844852 25146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf14701a02e64de4ad3f58eee3e55fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:08:58.847460 25229 tablet_bootstrap.cc:492] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193: Bootstrap starting.
I20260925 10:08:58.847918 25230 tablet_bootstrap.cc:492] T bf14701a02e64de4ad3f58eee3e55fba P 4a3641f4f5514331a4b6738b86beee72: Bootstrap starting.
I20260925 10:08:58.848088 25231 tablet_bootstrap.cc:492] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26: Bootstrap starting.
I20260925 10:08:58.848366 25229 tablet_bootstrap.cc:654] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:58.848592 25230 tablet_bootstrap.cc:654] T bf14701a02e64de4ad3f58eee3e55fba P 4a3641f4f5514331a4b6738b86beee72: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:58.848778 25231 tablet_bootstrap.cc:654] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26: Neither blocks nor log segments found. Creating new log.
I20260925 10:08:58.848948 25229 log.cc:824] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:58.849176 25230 log.cc:824] T bf14701a02e64de4ad3f58eee3e55fba P 4a3641f4f5514331a4b6738b86beee72: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:58.849352 25231 log.cc:824] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26: Log is configured to *not* fsync() on all Append() calls
I20260925 10:08:58.849779 25229 tablet_bootstrap.cc:492] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193: No bootstrap required, opened a new log
I20260925 10:08:58.849897 25229 ts_tablet_manager.cc:1397] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:08:58.849948 25230 tablet_bootstrap.cc:492] T bf14701a02e64de4ad3f58eee3e55fba P 4a3641f4f5514331a4b6738b86beee72: No bootstrap required, opened a new log
I20260925 10:08:58.850019 25230 ts_tablet_manager.cc:1397] T bf14701a02e64de4ad3f58eee3e55fba P 4a3641f4f5514331a4b6738b86beee72: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:08:58.850442 25231 tablet_bootstrap.cc:492] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26: No bootstrap required, opened a new log
I20260925 10:08:58.850510 25231 ts_tablet_manager.cc:1397] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:08:58.851522 25230 raft_consensus.cc:348] T bf14701a02e64de4ad3f58eee3e55fba P 4a3641f4f5514331a4b6738b86beee72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9894174acee14ea6953d8bec3d3dc193" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45141 } } peers { permanent_uuid: "4a3641f4f5514331a4b6738b86beee72" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34315 } } peers { permanent_uuid: "6d676fedadd64872b9eaf0d8eabdca26" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 35927 } }
I20260925 10:08:58.851683 25230 raft_consensus.cc:374] T bf14701a02e64de4ad3f58eee3e55fba P 4a3641f4f5514331a4b6738b86beee72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:58.851737 25230 raft_consensus.cc:729] T bf14701a02e64de4ad3f58eee3e55fba P 4a3641f4f5514331a4b6738b86beee72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a3641f4f5514331a4b6738b86beee72, State: Initialized, Role: FOLLOWER
I20260925 10:08:58.851897 25230 consensus_queue.cc:260] T bf14701a02e64de4ad3f58eee3e55fba P 4a3641f4f5514331a4b6738b86beee72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9894174acee14ea6953d8bec3d3dc193" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45141 } } peers { permanent_uuid: "4a3641f4f5514331a4b6738b86beee72" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34315 } } peers { permanent_uuid: "6d676fedadd64872b9eaf0d8eabdca26" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 35927 } }
I20260925 10:08:58.851981 25229 raft_consensus.cc:348] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9894174acee14ea6953d8bec3d3dc193" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45141 } } peers { permanent_uuid: "4a3641f4f5514331a4b6738b86beee72" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34315 } } peers { permanent_uuid: "6d676fedadd64872b9eaf0d8eabdca26" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 35927 } }
I20260925 10:08:58.852144 25229 raft_consensus.cc:374] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:58.852186 25229 raft_consensus.cc:729] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9894174acee14ea6953d8bec3d3dc193, State: Initialized, Role: FOLLOWER
I20260925 10:08:58.852208 25230 ts_tablet_manager.cc:1428] T bf14701a02e64de4ad3f58eee3e55fba P 4a3641f4f5514331a4b6738b86beee72: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:08:58.852329 24944 heartbeater.cc:503] Master 127.14.41.126:46521 was elected leader, sending a full tablet report...
I20260925 10:08:58.852337 25229 consensus_queue.cc:260] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9894174acee14ea6953d8bec3d3dc193" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45141 } } peers { permanent_uuid: "4a3641f4f5514331a4b6738b86beee72" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34315 } } peers { permanent_uuid: "6d676fedadd64872b9eaf0d8eabdca26" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 35927 } }
I20260925 10:08:58.852515 25231 raft_consensus.cc:348] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9894174acee14ea6953d8bec3d3dc193" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45141 } } peers { permanent_uuid: "4a3641f4f5514331a4b6738b86beee72" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34315 } } peers { permanent_uuid: "6d676fedadd64872b9eaf0d8eabdca26" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 35927 } }
I20260925 10:08:58.852675 25078 heartbeater.cc:503] Master 127.14.41.126:46521 was elected leader, sending a full tablet report...
I20260925 10:08:58.852685 25231 raft_consensus.cc:374] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:08:58.852730 25231 raft_consensus.cc:729] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d676fedadd64872b9eaf0d8eabdca26, State: Initialized, Role: FOLLOWER
I20260925 10:08:58.852747 25229 ts_tablet_manager.cc:1428] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:08:58.852856 25231 consensus_queue.cc:260] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9894174acee14ea6953d8bec3d3dc193" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45141 } } peers { permanent_uuid: "4a3641f4f5514331a4b6738b86beee72" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34315 } } peers { permanent_uuid: "6d676fedadd64872b9eaf0d8eabdca26" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 35927 } }
I20260925 10:08:58.853117 25231 ts_tablet_manager.cc:1428] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:08:58.853233 25212 heartbeater.cc:503] Master 127.14.41.126:46521 was elected leader, sending a full tablet report...
W20260925 10:08:58.889531 25079 tablet.cc:2369] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:08:58.972886 24945 tablet.cc:2369] T bf14701a02e64de4ad3f58eee3e55fba P 4a3641f4f5514331a4b6738b86beee72: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:08:59.057807 25213 tablet.cc:2369] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:08:59.238543 25236 raft_consensus.cc:482] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:59.238859 25236 raft_consensus.cc:504] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9894174acee14ea6953d8bec3d3dc193" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45141 } } peers { permanent_uuid: "4a3641f4f5514331a4b6738b86beee72" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34315 } } peers { permanent_uuid: "6d676fedadd64872b9eaf0d8eabdca26" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 35927 } }
I20260925 10:08:59.239398 25237 raft_consensus.cc:482] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:08:59.239574 25237 raft_consensus.cc:504] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9894174acee14ea6953d8bec3d3dc193" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45141 } } peers { permanent_uuid: "4a3641f4f5514331a4b6738b86beee72" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34315 } } peers { permanent_uuid: "6d676fedadd64872b9eaf0d8eabdca26" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 35927 } }
I20260925 10:08:59.239624 25236 leader_election.cc:288] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a3641f4f5514331a4b6738b86beee72 (127.14.41.65:34315), 6d676fedadd64872b9eaf0d8eabdca26 (127.14.41.67:35927)
I20260925 10:08:59.240623 25237 leader_election.cc:288] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9894174acee14ea6953d8bec3d3dc193 (127.14.41.66:45141), 4a3641f4f5514331a4b6738b86beee72 (127.14.41.65:34315)
I20260925 10:08:59.243474 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:59.239703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:42305 (local address 127.14.41.67:35927)
0925 10:08:59.239746 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:59.239748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:59.239758 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:08:59.239865 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:59.239867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:59.239877 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:59.239913 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:59.240198 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:59.240959 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:59.242495 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:59.243008 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:59.243055 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:59.243063 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:59.243349 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 10:08:59.243396 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:08:59.243606 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:59.240270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:45141 (local address 127.14.41.67:59931)
0925 10:08:59.240315 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:59.240324 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:59.240358 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:59.240626 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:59.240627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:59.240629 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:59.240685 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:59.240686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:59.241493 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:59.241493 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:59.242660 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:59.242663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:59.243386 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:59.243387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:59.243457 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:59.243461 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:08:59.243508 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 10:08:59.243546 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 10:08:59.243863 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:59.240460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:59931 (local address 127.14.41.66:45141)
0925 10:08:59.240496 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:59.240498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:59.240505 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:08:59.240526 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:59.240527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:59.240534 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:59.240586 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:59.240752 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:59.241414 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:59.242744 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:59.243284 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:59.243349 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:59.243356 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:59.243763 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 10:08:59.243803 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:08:59.244071 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:59.239488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:35927 (local address 127.14.41.66:42305)
0925 10:08:59.239693 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:59.239705 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:59.239767 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:59.240037 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:59.240038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:59.240039 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:59.240094 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:59.240096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:59.241048 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:59.241050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:59.242402 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:59.242405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:59.243195 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:59.243196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:59.243266 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:59.243269 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:08:59.243912 (+ 643us) client_negotiation.cc:240] Negotiation successful
0925 10:08:59.243984 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"spinlock_wait_cycles":12160,"thread_start_us":94,"threads_started":1}
I20260925 10:08:59.244138 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf14701a02e64de4ad3f58eee3e55fba" candidate_uuid: "6d676fedadd64872b9eaf0d8eabdca26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9894174acee14ea6953d8bec3d3dc193" is_pre_election: true
I20260925 10:08:59.244252 25032 raft_consensus.cc:2454] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d676fedadd64872b9eaf0d8eabdca26 in term 0.
I20260925 10:08:59.244388 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf14701a02e64de4ad3f58eee3e55fba" candidate_uuid: "9894174acee14ea6953d8bec3d3dc193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d676fedadd64872b9eaf0d8eabdca26" is_pre_election: true
I20260925 10:08:59.244438 25100 leader_election.cc:302] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d676fedadd64872b9eaf0d8eabdca26, 9894174acee14ea6953d8bec3d3dc193; no voters:
I20260925 10:08:59.244509 25166 raft_consensus.cc:2454] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9894174acee14ea6953d8bec3d3dc193 in term 0.
I20260925 10:08:59.244572 25237 raft_consensus.cc:2781] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:59.244612 25237 raft_consensus.cc:482] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:59.244642 25237 raft_consensus.cc:3037] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:59.244643 25241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:59.241049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:36081 (local address 127.14.41.65:34315)
0925 10:08:59.241163 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:59.241165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:59.241172 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:08:59.241193 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:59.241194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:59.241204 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:59.241271 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:59.241440 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:59.242151 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:59.243462 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:59.244132 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:59.244470 (+ 338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:59.244481 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:59.244499 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 10:08:59.244543 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":49,"threads_started":1}
I20260925 10:08:59.244690 24968 leader_election.cc:302] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d676fedadd64872b9eaf0d8eabdca26, 9894174acee14ea6953d8bec3d3dc193; no voters:
I20260925 10:08:59.245100 25228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:59.239601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:39421 (local address 127.14.41.65:34315)
0925 10:08:59.239653 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:59.239655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:59.239800 (+ 145us) server_negotiation.cc:407] Connection header received
0925 10:08:59.239864 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:59.239865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:59.239876 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:59.239995 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:08:59.240217 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:59.240996 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:59.242605 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:59.243208 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:59.243254 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:59.243262 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:59.245005 (+ 1743us) server_negotiation.cc:299] Negotiation successful
0925 10:08:59.245046 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:08:59.245369 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:59.239487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:34315 (local address 127.14.41.66:39421)
0925 10:08:59.239744 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:59.239752 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:59.239803 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:59.240003 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:59.240004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:59.240006 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:59.240094 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:59.240117 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:59.241101 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:59.241102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:59.242502 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:59.242505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:59.244801 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:59.244802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:59.244904 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:59.244909 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:08:59.245244 (+ 335us) client_negotiation.cc:240] Negotiation successful
0925 10:08:59.245290 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":191,"thread_start_us":94,"threads_started":1}
I20260925 10:08:59.245402 25240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:59.240515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:34315 (local address 127.14.41.67:36081)
0925 10:08:59.240850 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:59.240860 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:08:59.240908 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:08:59.241276 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:08:59.241277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:08:59.241279 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:08:59.241339 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:59.241340 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:59.242249 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:59.242250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:59.243372 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:08:59.243374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:08:59.244249 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:08:59.244251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:08:59.244329 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:59.244333 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:08:59.244365 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 10:08:59.244403 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":80,"threads_started":1}
I20260925 10:08:59.245581 25237 raft_consensus.cc:504] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9894174acee14ea6953d8bec3d3dc193" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45141 } } peers { permanent_uuid: "4a3641f4f5514331a4b6738b86beee72" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34315 } } peers { permanent_uuid: "6d676fedadd64872b9eaf0d8eabdca26" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 35927 } }
I20260925 10:08:59.245718 25237 leader_election.cc:288] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [CANDIDATE]: Term 1 election: Requested vote from peers 9894174acee14ea6953d8bec3d3dc193 (127.14.41.66:45141), 4a3641f4f5514331a4b6738b86beee72 (127.14.41.65:34315)
I20260925 10:08:59.245726 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf14701a02e64de4ad3f58eee3e55fba" candidate_uuid: "6d676fedadd64872b9eaf0d8eabdca26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a3641f4f5514331a4b6738b86beee72" is_pre_election: true
I20260925 10:08:59.245882 24898 raft_consensus.cc:2454] T bf14701a02e64de4ad3f58eee3e55fba P 4a3641f4f5514331a4b6738b86beee72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d676fedadd64872b9eaf0d8eabdca26 in term 0.
I20260925 10:08:59.245926 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf14701a02e64de4ad3f58eee3e55fba" candidate_uuid: "6d676fedadd64872b9eaf0d8eabdca26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a3641f4f5514331a4b6738b86beee72"
I20260925 10:08:59.245987 24897 raft_consensus.cc:3037] T bf14701a02e64de4ad3f58eee3e55fba P 4a3641f4f5514331a4b6738b86beee72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:59.246052 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf14701a02e64de4ad3f58eee3e55fba" candidate_uuid: "9894174acee14ea6953d8bec3d3dc193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a3641f4f5514331a4b6738b86beee72" is_pre_election: true
I20260925 10:08:59.246115 25236 raft_consensus.cc:2781] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:08:59.246200 25236 raft_consensus.cc:482] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:08:59.246261 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf14701a02e64de4ad3f58eee3e55fba" candidate_uuid: "6d676fedadd64872b9eaf0d8eabdca26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9894174acee14ea6953d8bec3d3dc193"
I20260925 10:08:59.246488 25236 raft_consensus.cc:3037] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:08:59.246940 24897 raft_consensus.cc:2454] T bf14701a02e64de4ad3f58eee3e55fba P 4a3641f4f5514331a4b6738b86beee72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d676fedadd64872b9eaf0d8eabdca26 in term 1.
I20260925 10:08:59.247108 25101 leader_election.cc:302] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a3641f4f5514331a4b6738b86beee72, 6d676fedadd64872b9eaf0d8eabdca26; no voters:
I20260925 10:08:59.247247 25237 raft_consensus.cc:2781] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:08:59.247300 25236 raft_consensus.cc:504] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9894174acee14ea6953d8bec3d3dc193" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45141 } } peers { permanent_uuid: "4a3641f4f5514331a4b6738b86beee72" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34315 } } peers { permanent_uuid: "6d676fedadd64872b9eaf0d8eabdca26" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 35927 } }
I20260925 10:08:59.247426 25237 raft_consensus.cc:686] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [term 1 LEADER]: Becoming Leader. State: Replica: 6d676fedadd64872b9eaf0d8eabdca26, State: Running, Role: LEADER
I20260925 10:08:59.247442 25032 raft_consensus.cc:2379] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6d676fedadd64872b9eaf0d8eabdca26 in current term 1: Already voted for candidate 9894174acee14ea6953d8bec3d3dc193 in this term.
I20260925 10:08:59.247536 25237 consensus_queue.cc:237] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9894174acee14ea6953d8bec3d3dc193" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45141 } } peers { permanent_uuid: "4a3641f4f5514331a4b6738b86beee72" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34315 } } peers { permanent_uuid: "6d676fedadd64872b9eaf0d8eabdca26" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 35927 } }
I20260925 10:08:59.247818 24897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf14701a02e64de4ad3f58eee3e55fba" candidate_uuid: "9894174acee14ea6953d8bec3d3dc193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a3641f4f5514331a4b6738b86beee72"
I20260925 10:08:59.247920 24897 raft_consensus.cc:2379] T bf14701a02e64de4ad3f58eee3e55fba P 4a3641f4f5514331a4b6738b86beee72 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9894174acee14ea6953d8bec3d3dc193 in current term 1: Already voted for candidate 6d676fedadd64872b9eaf0d8eabdca26 in this term.
I20260925 10:08:59.247970 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf14701a02e64de4ad3f58eee3e55fba" candidate_uuid: "9894174acee14ea6953d8bec3d3dc193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d676fedadd64872b9eaf0d8eabdca26"
I20260925 10:08:59.247821 25236 leader_election.cc:288] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193 [CANDIDATE]: Term 1 election: Requested vote from peers 4a3641f4f5514331a4b6738b86beee72 (127.14.41.65:34315), 6d676fedadd64872b9eaf0d8eabdca26 (127.14.41.67:35927)
I20260925 10:08:59.248193 24968 leader_election.cc:302] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9894174acee14ea6953d8bec3d3dc193; no voters: 4a3641f4f5514331a4b6738b86beee72, 6d676fedadd64872b9eaf0d8eabdca26
I20260925 10:08:59.248294 25236 raft_consensus.cc:2726] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 10:08:59.248358 24753 catalog_manager.cc:5495] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d676fedadd64872b9eaf0d8eabdca26 (127.14.41.67). New cstate: current_term: 1 leader_uuid: "6d676fedadd64872b9eaf0d8eabdca26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9894174acee14ea6953d8bec3d3dc193" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a3641f4f5514331a4b6738b86beee72" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d676fedadd64872b9eaf0d8eabdca26" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 35927 } health_report { overall_health: HEALTHY } } }
I20260925 10:08:59.283429 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:59.281062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40174 (local address 127.14.41.67:35927)
0925 10:08:59.281123 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 10:08:59.281125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:08:59.281134 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:08:59.281185 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:08:59.281186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:08:59.281204 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:08:59.281266 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:08:59.281395 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:59.281874 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:59.282619 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:08:59.282802 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:08:59.282856 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:08:59.282998 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:08:59.283102 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:08:59.283163 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:08:59.283271 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 10:08:59.283295 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 10:08:59.285353 25032 raft_consensus.cc:1261] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193 [term 1 FOLLOWER]: Refusing update from remote peer 6d676fedadd64872b9eaf0d8eabdca26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:59.285379 24897 raft_consensus.cc:1261] T bf14701a02e64de4ad3f58eee3e55fba P 4a3641f4f5514331a4b6738b86beee72 [term 1 FOLLOWER]: Refusing update from remote peer 6d676fedadd64872b9eaf0d8eabdca26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:08:59.285624 25237 consensus_queue.cc:1035] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9894174acee14ea6953d8bec3d3dc193" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:59.285792 25242 consensus_queue.cc:1035] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3641f4f5514331a4b6738b86beee72" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:08:59.288175 25245 mvcc.cc:205] Tried to move back new op lower bound from 7333195527310172160 to 7333195527158394880. Current Snapshot: MvccSnapshot[applied={T|T < 7333195527310172160}]
I20260925 10:08:59.288376 25246 mvcc.cc:205] Tried to move back new op lower bound from 7333195527310172160 to 7333195527158394880. Current Snapshot: MvccSnapshot[applied={T|T < 7333195527310172160}]
I20260925 10:08:59.288540 25247 mvcc.cc:205] Tried to move back new op lower bound from 7333195527310172160 to 7333195527158394880. Current Snapshot: MvccSnapshot[applied={T|T < 7333195527310172160}]
I20260925 10:08:59.290910 24753 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:59.291019 24753 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:08:59.291882 24753 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16d27b71e6114e32bdbace34439f2733: Sending DeleteTablet for 3 replicas of tablet bf14701a02e64de4ad3f58eee3e55fba
I20260925 10:08:59.292308 25146 tablet_service.cc:1514] Processing DeleteTablet for tablet bf14701a02e64de4ad3f58eee3e55fba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:59 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:40158
I20260925 10:08:59.292366 25010 tablet_service.cc:1514] Processing DeleteTablet for tablet bf14701a02e64de4ad3f58eee3e55fba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:59 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:47052
I20260925 10:08:59.292605 24878 tablet_service.cc:1514] Processing DeleteTablet for tablet bf14701a02e64de4ad3f58eee3e55fba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:08:59 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:56524
I20260925 10:08:59.292843 25257 tablet_replica.cc:330] T bf14701a02e64de4ad3f58eee3e55fba P 4a3641f4f5514331a4b6738b86beee72: stopping tablet replica
I20260925 10:08:59.292878 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 24815
I20260925 10:08:59.298807 25256 tablet_replica.cc:330] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193: stopping tablet replica
W20260925 10:08:59.298781 24739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.41.65:34315 (error 108)
I20260925 10:08:59.298964 25256 raft_consensus.cc:2229] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:08:59.299099 25255 tablet_replica.cc:330] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26: stopping tablet replica
I20260925 10:08:59.299100 25256 raft_consensus.cc:2258] T bf14701a02e64de4ad3f58eee3e55fba P 9894174acee14ea6953d8bec3d3dc193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:59.299396 25255 raft_consensus.cc:2229] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:08:59.299413 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 24949
I20260925 10:08:59.300163 25255 raft_consensus.cc:2258] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:08:59.300675 25255 ts_tablet_manager.cc:1905] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:08:59.302162 25255 ts_tablet_manager.cc:1918] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:08:59.302258 25255 log.cc:1196] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/bf14701a02e64de4ad3f58eee3e55fba
I20260925 10:08:59.302572 25255 ts_tablet_manager.cc:1939] T bf14701a02e64de4ad3f58eee3e55fba P 6d676fedadd64872b9eaf0d8eabdca26: Deleting consensus metadata
I20260925 10:08:59.303002 24740 catalog_manager.cc:4841] TS 6d676fedadd64872b9eaf0d8eabdca26 (127.14.41.67:35927): tablet bf14701a02e64de4ad3f58eee3e55fba (table test-table [id=ccbe6a79568f4777acc2eb592185cb25]) successfully deleted
I20260925 10:08:59.303123 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:59.302982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:34315 (local address 127.0.0.1:56530)
0925 10:08:59.303033 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:59.303088 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.65:34315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 10:08:59.303233 24739 catalog_manager.cc:4584] TS 4a3641f4f5514331a4b6738b86beee72 (127.14.41.65:34315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf14701a02e64de4ad3f58eee3e55fba: Network error: Client connection negotiation failed: client connection to 127.14.41.65:34315: connect: Connection refused (error 111)
I20260925 10:08:59.307379 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:08:59.307258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:45141 (local address 127.0.0.1:47056)
0925 10:08:59.307312 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 10:08:59.307347 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.66:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 10:08:59.307370 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 25083
I20260925 10:08:59.312299 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 24721
2026-09-25T10:08:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2383 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:08:59 dist-test-slave-s6tm krb5kdc[25261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:08:59 dist-test-slave-s6tm krb5kdc[25261](info): set up 2 sockets
Sep 25 10:08:59 dist-test-slave-s6tm krb5kdc[25261](info): commencing operation
krb5kdc: starting...
W20260925 10:09:00.406406 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:09:00 dist-test-slave-s6tm krb5kdc[25261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:09:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:09:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.41.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.41.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.41.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/rangeradmin_127.14.41.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.41.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/rangeradmin_127.14.41.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.41.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.41.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.41.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/rangerlookup_127.14.41.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.41.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/rangerlookup_127.14.41.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.41.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.41.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.41.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/HTTP_127.14.41.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.41.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/HTTP_127.14.41.84@KRBTEST.COM.keytab.
I20260925 10:09:00.574497 14501 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskdDYjLr/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/postgres -l logfile start
2026-09-25 10:09:03.226 UTC [25291] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 10:09:03.226 UTC [25291] LOG: listening on IPv4 address "127.14.41.84", port 47829
2026-09-25 10:09:03.230 UTC [25291] LOG: listening on Unix socket "/tmp/.s.PGSQL.47829"
2026-09-25 10:09:03.253 UTC [25294] LOG: database system was shut down at 2026-09-25 10:09:01 UTC
2026-09-25 10:09:03.257 UTC [25291] LOG: database system is ready to accept connections
I20260925 10:09:04.278192 14501 mini_postgres.cc:96] Postgres bound to 47829
2026-09-25 10:09:04.285 UTC [25302] FATAL: database "slave" does not exist
127.14.41.84:47829 - accepting connections
I20260925 10:09:04.285624 14501 mini_ranger.cc:161] Starting Ranger...
I20260925 10:09:04.300969 14501 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 10:09:04.831390 14501 mini_ranger.cc:87] Created ranger Postgres database
I20260925 10:09:04.831462 14501 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ranger-admin
2026-09-25 10:09:05,167 [I] DB FLAVOR :POSTGRES
2026-09-25 10:09:05,168 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:09:05,168 [I] Checking connection..
2026-09-25 10:09:05,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 10:09:05,434 [I] Checking connection passed.
2026-09-25 10:09:05,435 [I] --------- Verifying version history table ---------
2026-09-25 10:09:05,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:09:05,709 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:09:05,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:09:05,970 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:09:05,970 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 10:09:05,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 10:09:06.235 UTC [25366] WARNING: there is no transaction in progress
2026-09-25 10:09:06,242 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 10:09:06,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:09:06,515 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 10:09:06,516 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:09:06,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 10:09:06,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 10:09:07,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-s6tm.c.gcp-upstream.internal','N') ;"
2026-09-25 10:09:07,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:09:07,475 [I] Table x_portal_user does not exist in database ranger
2026-09-25 10:09:07,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:09:07,708 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 10:09:07,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:09:07,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:09:08,241 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 10:09:08,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 10:09:09.287 UTC [25474] WARNING: there is no transaction in progress
2026-09-25 10:09:09.305 UTC [25474] WARNING: there is no transaction in progress
2026-09-25 10:09:09.323 UTC [25474] WARNING: there is no transaction in progress
2026-09-25 10:09:09.338 UTC [25474] WARNING: there is no transaction in progress
2026-09-25 10:09:09.354 UTC [25474] WARNING: there is no transaction in progress
2026-09-25 10:09:09.375 UTC [25474] WARNING: there is no transaction in progress
2026-09-25 10:09:09.432 UTC [25474] WARNING: there is no transaction in progress
2026-09-25 10:09:09.439 UTC [25474] WARNING: there is no transaction in progress
2026-09-25 10:09:09.454 UTC [25474] WARNING: there is no transaction in progress
2026-09-25 10:09:09.466 UTC [25474] WARNING: there is no transaction in progress
2026-09-25 10:09:09.479 UTC [25474] WARNING: there is no transaction in progress
2026-09-25 10:09:09.495 UTC [25474] WARNING: there is no transaction in progress
2026-09-25 10:09:09.506 UTC [25474] WARNING: there is no transaction in progress
2026-09-25 10:09:09.513 UTC [25474] WARNING: there is no transaction in progress
2026-09-25 10:09:09.521 UTC [25474] WARNING: there is no transaction in progress
2026-09-25 10:09:09.529 UTC [25474] WARNING: there is no transaction in progress
2026-09-25 10:09:10,250 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 10:09:10,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 10:09:10,505 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 10:09:10,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:09:10,760 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 10:09:10,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:09:11,019 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 10:09:11,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:09:11,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:09:11,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-s6tm.c.gcp-upstream.internal';"
2026-09-25 10:09:11,748 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 10:09:11,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 10:09:11,987 [I] DB_PATCHES have already been applied
I20260925 10:09:11.993069 14501 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ranger-admin:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskdDYjLr/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 10:09:11.993142 14501 mini_ranger.cc:193] Using host: 127.14.41.84
I20260925 10:09:11.996359 14501 mini_ranger.cc:227] Ranger admin URL: http://127.14.41.84:33223
Sep 25, 2026 10:09:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:09:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:09:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:09:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:09:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:09:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:09:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 10:09:13.104049 14501 mini_ranger.cc:160] Time spent starting Ranger: real 8.818s user 0.005s sys 0.005s
Sep 25, 2026 10:09:33 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:09:33 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 10:09:33 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:09:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:09:33 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:09:34 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 10:09:35.453186 14501 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.41.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.41.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.41.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/rangerkms_127.14.41.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.41.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/rangerkms_127.14.41.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.41.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.41.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.41.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/HTTP_127.14.41.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.41.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/HTTP_127.14.41.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 10:09:35 dist-test-slave-s6tm krb5kdc[25261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330975, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 10:09:35.552069 14501 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 10:09:35.562135 14501 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 10:09:36.549060 14501 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 10:09:36.549163 14501 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ranger-kms
I20260925 10:09:36.549181 14501 mini_ranger_kms.cc:227] Using postgres at 127.14.41.84:47829
2026-09-25 10:09:36,822 [I] DB FLAVOR :POSTGRES
2026-09-25 10:09:36,823 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:09:36,823 [I] Checking connection
2026-09-25 10:09:36,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 10:09:37,067 [I] connection success
2026-09-25 10:09:37,067 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 10:09:37,067 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 10:09:37,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 10:09:37,304 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 10:09:37,304 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:09:37,305 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 10:09:37,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:47829/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 10:09:37,583 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 10:09:38.106379 14501 mini_ranger_kms.cc:310] Created kms service
I20260925 10:09:38.304219 14501 mini_ranger_kms.cc:326] Created kudu user
I20260925 10:09:38.346551 14501 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 10:09:38 dist-test-slave-s6tm krb5kdc[25261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330975, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.41.84@KRBTEST.COM
I20260925 10:09:38.793975 14501 mini_ranger_kms.cc:384] Added ranger policy
I20260925 10:09:38.794123 14501 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ranger-kms:/tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 10:09:38.794152 14501 mini_ranger_kms.cc:240] Using host: 127.14.41.84
I20260925 10:09:38.797816 14501 mini_ranger_kms.cc:276] Ranger KMS PID: 25655
I20260925 10:09:38.797955 14501 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.41.84:45129
Sep 25, 2026 10:09:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:09:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:09:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:09:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:09:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:09:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 10:09:41.146750 14501 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.595s user 0.017s sys 0.146s
I20260925 10:09:41.146867 14501 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 10:09:41.146962 14501 mini_ranger_kms.cc:401] 127.14.41.84:45129/kms/v1/keys
Sep 25 10:09:41 dist-test-slave-s6tm krb5kdc[25261](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790330981, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.41.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 10:09:41 dist-test-slave-s6tm krb5kdc[25261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330981, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.41.84@KRBTEST.COM for HTTP/127.14.41.84@KRBTEST.COM
Sep 25 10:09:41 dist-test-slave-s6tm krb5kdc[25261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330981, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.41.84@KRBTEST.COM for HTTP/127.14.41.84@KRBTEST.COM
Sep 25 10:09:44 dist-test-slave-s6tm krb5kdc[25261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:09:44.359375 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:34621
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:41723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:34621
--ranger_config_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.41.84:45129/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:09:44.442200 25705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:09:44.442423 25705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:09:44.442445 25705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:09:44.443630 25705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:09:44.443681 25705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:09:44.443696 25705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:09:44.443713 25705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:09:44.443727 25705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:09:44.445397 25705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:41723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.41.84:45129/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:34621
--ranger_config_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:34621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:09:44.445659 25705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:09:44.445919 25705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:09:44.452085 25711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:09:44.452142 25705 server_base.cc:1034] running on GCE node
W20260925 10:09:44.452215 25713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:09:44.452107 25710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:09:44.452663 25705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:09:44.453012 25705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:09:44.454203 25705 hybrid_clock.cc:648] HybridClock initialized: now 1790330984454186 us; error 32 us; skew 500 ppm
Sep 25 10:09:44 dist-test-slave-s6tm krb5kdc[25261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330984, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:09:44.462266 25705 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:09:44.462883 25705 webserver.cc:460] Webserver started at http://127.14.41.126:45257/ using document root <none> and password file <none>
I20260925 10:09:44.463138 25705 fs_manager.cc:359] Metadata directory not provided
I20260925 10:09:44.463182 25705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:09:44.463346 25705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:09:44 dist-test-slave-s6tm krb5kdc[25261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330984, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for HTTP/127.14.41.84@KRBTEST.COM
I20260925 10:09:44.522105 25705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "5c352fe0c3d64c47b0bedd936db68b97"
format_stamp: "Formatted at 2026-09-25 10:09:44 on dist-test-slave-s6tm"
server_key: "bb926b79d676a68daa9fb3bcdd32d9ce"
server_key_iv: "4b53b9defc0d88025a3f6450e7e03a3a"
server_key_version: "kuduclusterkey@0"
I20260925 10:09:44.522614 25705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "5c352fe0c3d64c47b0bedd936db68b97"
format_stamp: "Formatted at 2026-09-25 10:09:44 on dist-test-slave-s6tm"
server_key: "bb926b79d676a68daa9fb3bcdd32d9ce"
server_key_iv: "4b53b9defc0d88025a3f6450e7e03a3a"
server_key_version: "kuduclusterkey@0"
I20260925 10:09:44.524535 25705 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 10:09:44.539283 25720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:09:44.539570 25705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:09:44.539690 25705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "5c352fe0c3d64c47b0bedd936db68b97"
format_stamp: "Formatted at 2026-09-25 10:09:44 on dist-test-slave-s6tm"
server_key: "bb926b79d676a68daa9fb3bcdd32d9ce"
server_key_iv: "4b53b9defc0d88025a3f6450e7e03a3a"
server_key_version: "kuduclusterkey@0"
I20260925 10:09:44.539809 25705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:09:44.566226 25705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:09:44.568712 25705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:09:44.568882 25705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:09:44.574044 25705 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:34621
I20260925 10:09:44.574064 25772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:34621 every 8 connection(s)
I20260925 10:09:44.574543 25705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:09:44.575315 25773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:09:44.576120 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 25705
I20260925 10:09:44.576246 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:09:44.576432 14501 external_mini_cluster.cc:1428] Setting key 91b84153fc5c8ca780b59996f718f3e4
I20260925 10:09:44.578755 25773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97: Bootstrap starting.
I20260925 10:09:44.579457 25773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97: Neither blocks nor log segments found. Creating new log.
I20260925 10:09:44.579761 25773 log.cc:824] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97: Log is configured to *not* fsync() on all Append() calls
I20260925 10:09:44.580574 25773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97: No bootstrap required, opened a new log
I20260925 10:09:44.582929 25773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c352fe0c3d64c47b0bedd936db68b97" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 34621 } }
I20260925 10:09:44.583109 25773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:09:44.583148 25773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c352fe0c3d64c47b0bedd936db68b97, State: Initialized, Role: FOLLOWER
I20260925 10:09:44.583343 25773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c352fe0c3d64c47b0bedd936db68b97" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 34621 } }
I20260925 10:09:44.583441 25773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:09:44.583508 25773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:09:44.583570 25773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:09:44.584295 25773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c352fe0c3d64c47b0bedd936db68b97" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 34621 } }
I20260925 10:09:44.584417 25773 leader_election.cc:302] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c352fe0c3d64c47b0bedd936db68b97; no voters:
Sep 25 10:09:44 dist-test-slave-s6tm krb5kdc[25261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:09:44.584616 25773 leader_election.cc:288] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:09:44.584641 25778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:09:44.584895 25773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:09:44.585825 25778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97 [term 1 LEADER]: Becoming Leader. State: Replica: 5c352fe0c3d64c47b0bedd936db68b97, State: Running, Role: LEADER
I20260925 10:09:44.585949 25778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c352fe0c3d64c47b0bedd936db68b97" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 34621 } }
I20260925 10:09:44.586691 25773 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 10:09:44.586655 25779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c352fe0c3d64c47b0bedd936db68b97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c352fe0c3d64c47b0bedd936db68b97" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 34621 } } }
I20260925 10:09:44.586768 25779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97 [sys.catalog]: This master's current role is: LEADER
I20260925 10:09:44.586895 25778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c352fe0c3d64c47b0bedd936db68b97. Latest consensus state: current_term: 1 leader_uuid: "5c352fe0c3d64c47b0bedd936db68b97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c352fe0c3d64c47b0bedd936db68b97" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 34621 } } }
I20260925 10:09:44.588992 25778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97 [sys.catalog]: This master's current role is: LEADER
I20260925 10:09:44.589353 25792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:09:44.591434 25792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:09:44.592636 25776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:09:44.577015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54894 (local address 127.14.41.126:34621)
0925 10:09:44.577217 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:09:44.577229 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:09:44.577266 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:09:44.577812 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:09:44.577822 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:09:44.577891 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:09:44.578055 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:09:44.578858 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:44.580255 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:44.581315 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:44.582113 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:44.582239 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:09:44.585265 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:09:44.585270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:09:44.585270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:09:44.585283 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:09:44.590115 (+ 4832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:09:44.590671 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:09:44.590675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:09:44.590675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:09:44.590944 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:09:44.591178 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:09:44.591179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:09:44.591179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:09:44.591650 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:09:44.591849 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 10:09:44.592002 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 10:09:44.592277 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
I20260925 10:09:44.597108 25792 catalog_manager.cc:1347] Generated new cluster ID: 751276e8aeb14010b5de545d88cda885
I20260925 10:09:44.597168 25792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:09:44.613653 25792 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:09:44.614490 25792 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:09:44.630235 25792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97: Generated new TSK 0
I20260925 10:09:44.630612 25792 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
I20260925 10:09:45.372991 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.41.84:45129/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:34621
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:41723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:09:45.483706 25818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:09:45.484026 25818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:09:45.484097 25818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:09:45.485445 25818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:09:45.485489 25818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:09:45.485520 25818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:09:45.487336 25818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:41723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.41.84:45129/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:34621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:09:45.487610 25818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:09:45.487856 25818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:09:45.493332 25824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:09:45.493355 25823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:09:45.493443 25826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:09:45.493532 25818 server_base.cc:1034] running on GCE node
I20260925 10:09:45.493839 25818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:09:45.494149 25818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:09:45.495354 25818 hybrid_clock.cc:648] HybridClock initialized: now 1790330985495330 us; error 32 us; skew 500 ppm
Sep 25 10:09:45 dist-test-slave-s6tm krb5kdc[25261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330985, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:09:45.498728 25818 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:09:45.499292 25818 webserver.cc:460] Webserver started at http://127.14.41.65:34725/ using document root <none> and password file <none>
I20260925 10:09:45.499560 25818 fs_manager.cc:359] Metadata directory not provided
I20260925 10:09:45.499624 25818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:09:45.499809 25818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:09:45 dist-test-slave-s6tm krb5kdc[25261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330985, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for HTTP/127.14.41.84@KRBTEST.COM
I20260925 10:09:45.514833 25818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "db95a542652546ac83c760b59ef84def"
format_stamp: "Formatted at 2026-09-25 10:09:45 on dist-test-slave-s6tm"
server_key: "ec59bc1de31d80f57206d520954e40bf"
server_key_iv: "d9d12a79f8a65e73e8f90050bbbbe68e"
server_key_version: "kuduclusterkey@0"
I20260925 10:09:45.515388 25818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "db95a542652546ac83c760b59ef84def"
format_stamp: "Formatted at 2026-09-25 10:09:45 on dist-test-slave-s6tm"
server_key: "ec59bc1de31d80f57206d520954e40bf"
server_key_iv: "d9d12a79f8a65e73e8f90050bbbbe68e"
server_key_version: "kuduclusterkey@0"
I20260925 10:09:45.520210 25818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.002s
I20260925 10:09:45.536923 25835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:09:45.537137 25818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:09:45.537233 25818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "db95a542652546ac83c760b59ef84def"
format_stamp: "Formatted at 2026-09-25 10:09:45 on dist-test-slave-s6tm"
server_key: "ec59bc1de31d80f57206d520954e40bf"
server_key_iv: "d9d12a79f8a65e73e8f90050bbbbe68e"
server_key_version: "kuduclusterkey@0"
I20260925 10:09:45.537350 25818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:09:45.550822 25818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:09:45.553567 25818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:09:45.553805 25818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:09:45.554124 25818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:09:45.554558 25818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:09:45.554670 25818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:09:45.554774 25818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:09:45.554852 25818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:09:45.562690 25818 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:38207
I20260925 10:09:45.562948 25948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:38207 every 8 connection(s)
I20260925 10:09:45.563215 25818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
Sep 25 10:09:45 dist-test-slave-s6tm krb5kdc[25261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330985, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:09:45.573066 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 25818
I20260925 10:09:45.573190 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:09:45.573381 14501 external_mini_cluster.cc:1428] Setting key c6739637c937aadf582cff0abf646a95
I20260925 10:09:45.579668 25951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:09:45.564077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:58429 (local address 127.14.41.126:34621)
0925 10:09:45.564224 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 10:09:45.564226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:09:45.564417 (+ 191us) server_negotiation.cc:407] Connection header received
0925 10:09:45.565103 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:09:45.565104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:09:45.565120 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:09:45.565199 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:09:45.565978 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:45.566755 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:45.567830 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:45.568126 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:45.568214 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:09:45.570810 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:09:45.570811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:09:45.570812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:09:45.570821 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:09:45.572829 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:09:45.578415 (+ 5586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:09:45.578417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:09:45.578417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:09:45.578501 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:09:45.578837 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:09:45.578838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:09:45.578838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:09:45.578989 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:09:45.579033 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 10:09:45.579434 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 10:09:45.579552 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260925 10:09:45.580547 25952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:09:45.564119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:34621 (local address 127.14.41.65:58429)
0925 10:09:45.564290 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 10:09:45.564333 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:09:45.564971 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:09:45.565279 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:09:45.565284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:09:45.565559 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:09:45.565858 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:09:45.565865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:09:45.566868 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:09:45.566869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:09:45.567730 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:09:45.567733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:09:45.568227 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:09:45.568228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:09:45.568368 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:09:45.568959 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:09:45.568967 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:09:45.570686 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:09:45.573619 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:09:45.573628 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:09:45.573631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:09:45.578289 (+ 4658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:09:45.578602 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:09:45.578603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:09:45.578603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:09:45.578761 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:09:45.579079 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:09:45.579086 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:09:45.579264 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 10:09:45.580088 (+ 824us) client_negotiation.cc:240] Negotiation successful
0925 10:09:45.580291 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260925 10:09:45.581293 25949 heartbeater.cc:347] Connected to a master server at 127.14.41.126:34621
I20260925 10:09:45.581390 25949 heartbeater.cc:464] Registering TS with master...
I20260925 10:09:45.581591 25949 heartbeater.cc:511] Master 127.14.41.126:34621 requested a full tablet report, sending...
I20260925 10:09:45.582175 25737 ts_manager.cc:194] Registered new tserver with Master: db95a542652546ac83c760b59ef84def (127.14.41.65:38207)
I20260925 10:09:45.583101 25737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:58429
WARNING: no policy specified for kudu/127.14.41.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
I20260925 10:09:45.620328 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.41.84:45129/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:34621
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:41723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:09:45 dist-test-slave-s6tm krb5kdc[25261](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790330985, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 10:09:45.756524 25955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:09:45.756901 25955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:09:45.756979 25955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:09:45.758867 25955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:09:45.759001 25955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:09:45.759105 25955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:09:45.761413 25955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:41723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.41.84:45129/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:34621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:09:45.761926 25955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:09:45.762247 25955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:09:45.769377 25965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:09:45.769678 25963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:09:45.769788 25962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:09:45.771095 25955 server_base.cc:1034] running on GCE node
I20260925 10:09:45.771454 25955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:09:45.771793 25955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:09:45.779151 25955 hybrid_clock.cc:648] HybridClock initialized: now 1790330985779103 us; error 56 us; skew 500 ppm
Sep 25 10:09:45 dist-test-slave-s6tm krb5kdc[25261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330985, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:09:45.782904 25955 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:09:45.783510 25955 webserver.cc:460] Webserver started at http://127.14.41.66:46171/ using document root <none> and password file <none>
I20260925 10:09:45.783798 25955 fs_manager.cc:359] Metadata directory not provided
I20260925 10:09:45.783855 25955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:09:45.784008 25955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:09:45 dist-test-slave-s6tm krb5kdc[25261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330985, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for HTTP/127.14.41.84@KRBTEST.COM
I20260925 10:09:45.805867 25955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "1ddc0c78407d40ba9d7c967fc71e99c9"
format_stamp: "Formatted at 2026-09-25 10:09:45 on dist-test-slave-s6tm"
server_key: "9b3bdfb98f611976e3fc3350c446f3d8"
server_key_iv: "42a25a49b1d45ec282ea7f4577be1f50"
server_key_version: "kuduclusterkey@0"
I20260925 10:09:45.806466 25955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "1ddc0c78407d40ba9d7c967fc71e99c9"
format_stamp: "Formatted at 2026-09-25 10:09:45 on dist-test-slave-s6tm"
server_key: "9b3bdfb98f611976e3fc3350c446f3d8"
server_key_iv: "42a25a49b1d45ec282ea7f4577be1f50"
server_key_version: "kuduclusterkey@0"
I20260925 10:09:45.812522 25955 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.000s
I20260925 10:09:45.831429 25972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:09:45.831632 25955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:09:45.831735 25955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "1ddc0c78407d40ba9d7c967fc71e99c9"
format_stamp: "Formatted at 2026-09-25 10:09:45 on dist-test-slave-s6tm"
server_key: "9b3bdfb98f611976e3fc3350c446f3d8"
server_key_iv: "42a25a49b1d45ec282ea7f4577be1f50"
server_key_version: "kuduclusterkey@0"
I20260925 10:09:45.831844 25955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:09:45.850689 25955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:09:45.853327 25955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:09:45.853541 25955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:09:45.853794 25955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:09:45.854218 25955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:09:45.854307 25955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:09:45.854414 25955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:09:45.854475 25955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:09:45.861685 25955 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:38151
I20260925 10:09:45.862217 25955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:09:45.864411 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 25955
I20260925 10:09:45.864522 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:09:45.864749 14501 external_mini_cluster.cc:1428] Setting key b111f593a54b335cc9d6197aee6cd9f2
I20260925 10:09:45.863713 26086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:38151 every 8 connection(s)
Sep 25 10:09:45 dist-test-slave-s6tm krb5kdc[25261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330985, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:09:45.881376 25951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:09:45.864979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:37333 (local address 127.14.41.126:34621)
0925 10:09:45.865036 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 10:09:45.865038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:09:45.868265 (+ 3227us) server_negotiation.cc:407] Connection header received
0925 10:09:45.868960 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:09:45.868962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:09:45.868977 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:09:45.869008 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:09:45.869893 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:45.870685 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:45.873189 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:45.873485 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:45.873527 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:09:45.877876 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:09:45.877878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:09:45.877879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:09:45.877888 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:09:45.879690 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:09:45.880242 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:09:45.880243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:09:45.880244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:09:45.880307 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:09:45.880583 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:09:45.880583 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:09:45.880584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:09:45.880734 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:09:45.880785 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 10:09:45.881173 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 10:09:45.881275 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:09:45.881915 26089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:09:45.864590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:34621 (local address 127.14.41.66:37333)
0925 10:09:45.868113 (+ 3523us) negotiation.cc:107] Waiting for socket to connect
0925 10:09:45.868173 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 10:09:45.868834 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:09:45.869222 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:09:45.869227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:09:45.869476 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:09:45.869757 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:09:45.869762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:09:45.871525 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:09:45.871526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:09:45.873083 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:09:45.873086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:09:45.873617 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:09:45.873618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:09:45.873757 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:09:45.876072 (+ 2315us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:09:45.876084 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:09:45.877756 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:09:45.879797 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:09:45.879804 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:09:45.879807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:09:45.880152 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:09:45.880376 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:09:45.880377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:09:45.880377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:09:45.880512 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:09:45.880819 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:09:45.880824 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:09:45.880999 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 10:09:45.881480 (+ 481us) client_negotiation.cc:240] Negotiation successful
0925 10:09:45.881672 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3466,"thread_start_us":77,"threads_started":1}
I20260925 10:09:45.882694 26087 heartbeater.cc:347] Connected to a master server at 127.14.41.126:34621
I20260925 10:09:45.882812 26087 heartbeater.cc:464] Registering TS with master...
I20260925 10:09:45.883040 26087 heartbeater.cc:511] Master 127.14.41.126:34621 requested a full tablet report, sending...
I20260925 10:09:45.883451 25737 ts_manager.cc:194] Registered new tserver with Master: 1ddc0c78407d40ba9d7c967fc71e99c9 (127.14.41.66:38151)
I20260925 10:09:45.883966 25737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:37333
WARNING: no policy specified for kudu/127.14.41.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
I20260925 10:09:45.909189 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.41.84:45129/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:34621
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:41723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:09:46.032516 26092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:09:46.032902 26092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:09:46.032995 26092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:09:46.034910 26092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:09:46.035187 26092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:09:46.035295 26092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:09:46.037794 26092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:41723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.41.84:45129/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:34621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:09:46.038331 26092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:09:46.038661 26092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:09:46.045488 26098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:09:46.045895 26097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:09:46.046136 26100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:09:46.047737 26092 server_base.cc:1034] running on GCE node
I20260925 10:09:46.047988 26092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:09:46.048274 26092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:09:46.049633 26092 hybrid_clock.cc:648] HybridClock initialized: now 1790330986049545 us; error 99 us; skew 500 ppm
Sep 25 10:09:46 dist-test-slave-s6tm krb5kdc[25261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330986, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:09:46.053153 26092 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:09:46.053694 26092 webserver.cc:460] Webserver started at http://127.14.41.67:41947/ using document root <none> and password file <none>
I20260925 10:09:46.053954 26092 fs_manager.cc:359] Metadata directory not provided
I20260925 10:09:46.054004 26092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:09:46.054145 26092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:09:46 dist-test-slave-s6tm krb5kdc[25261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330986, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for HTTP/127.14.41.84@KRBTEST.COM
I20260925 10:09:46.072026 26092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "93ded2dc783f4d56b64278476e69f1a5"
format_stamp: "Formatted at 2026-09-25 10:09:46 on dist-test-slave-s6tm"
server_key: "575c9218d043d55199db46b4fd5e07db"
server_key_iv: "81b27e6136c33021ff0351b7f84a4601"
server_key_version: "kuduclusterkey@0"
I20260925 10:09:46.072641 26092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "93ded2dc783f4d56b64278476e69f1a5"
format_stamp: "Formatted at 2026-09-25 10:09:46 on dist-test-slave-s6tm"
server_key: "575c9218d043d55199db46b4fd5e07db"
server_key_iv: "81b27e6136c33021ff0351b7f84a4601"
server_key_version: "kuduclusterkey@0"
I20260925 10:09:46.076752 26092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.002s
I20260925 10:09:46.095203 26107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:09:46.096035 26092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:09:46.096159 26092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "93ded2dc783f4d56b64278476e69f1a5"
format_stamp: "Formatted at 2026-09-25 10:09:46 on dist-test-slave-s6tm"
server_key: "575c9218d043d55199db46b4fd5e07db"
server_key_iv: "81b27e6136c33021ff0351b7f84a4601"
server_key_version: "kuduclusterkey@0"
I20260925 10:09:46.096263 26092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:09:46.116444 26092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:09:46.119143 26092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:09:46.119326 26092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:09:46.119629 26092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:09:46.120189 26092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:09:46.120285 26092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:09:46.120357 26092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:09:46.120409 26092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:09:46.129099 26092 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:39665
I20260925 10:09:46.129599 26092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:09:46.130259 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 26092
I20260925 10:09:46.130342 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:09:46.130522 14501 external_mini_cluster.cc:1428] Setting key 7d76b832fa69ff7bb3f16c9ed7742df1
I20260925 10:09:46.146067 26220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:39665 every 8 connection(s)
Sep 25 10:09:46 dist-test-slave-s6tm krb5kdc[25261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330986, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:09:46.174657 25951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:09:46.146656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:55181 (local address 127.14.41.126:34621)
0925 10:09:46.146702 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:09:46.146705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:09:46.152620 (+ 5915us) server_negotiation.cc:407] Connection header received
0925 10:09:46.156098 (+ 3478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:09:46.156100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:09:46.156115 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:09:46.156147 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:09:46.157127 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.157919 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.160859 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.161149 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.161201 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:09:46.170414 (+ 9213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:09:46.170416 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:09:46.170416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:09:46.170426 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:09:46.172511 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:09:46.173406 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:09:46.173407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:09:46.173408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:09:46.173468 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:09:46.173776 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:09:46.173777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:09:46.173777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:09:46.173922 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:09:46.173969 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 10:09:46.174444 (+ 475us) server_negotiation.cc:299] Negotiation successful
0925 10:09:46.174542 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:09:46.175513 26223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:09:46.146338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:34621 (local address 127.14.41.67:55181)
0925 10:09:46.152398 (+ 6060us) negotiation.cc:107] Waiting for socket to connect
0925 10:09:46.152477 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0925 10:09:46.153254 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:09:46.156392 (+ 3138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:09:46.156398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:09:46.156697 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:09:46.157010 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:09:46.157016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.158053 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.158053 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:09:46.159393 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:09:46.159396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.161305 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.161306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:09:46.161468 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:09:46.162163 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:09:46.162175 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:09:46.170282 (+ 8107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:09:46.172927 (+ 2645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:09:46.172938 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:09:46.172941 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:09:46.173309 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:09:46.173535 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:09:46.173536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:09:46.173536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:09:46.173691 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:09:46.174035 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:09:46.174043 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:09:46.174228 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 10:09:46.175019 (+ 791us) client_negotiation.cc:240] Negotiation successful
0925 10:09:46.175242 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5991,"thread_start_us":115,"threads_started":1}
I20260925 10:09:46.176301 26221 heartbeater.cc:347] Connected to a master server at 127.14.41.126:34621
I20260925 10:09:46.176441 26221 heartbeater.cc:464] Registering TS with master...
I20260925 10:09:46.176640 26221 heartbeater.cc:511] Master 127.14.41.126:34621 requested a full tablet report, sending...
I20260925 10:09:46.177068 25737 ts_manager.cc:194] Registered new tserver with Master: 93ded2dc783f4d56b64278476e69f1a5 (127.14.41.67:39665)
I20260925 10:09:46.177551 25737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:55181
I20260925 10:09:46.178993 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:09:46.187645 25951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:09:46.180512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54902 (local address 127.14.41.126:34621)
0925 10:09:46.180610 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 10:09:46.180613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:09:46.180656 (+ 43us) server_negotiation.cc:407] Connection header received
0925 10:09:46.180733 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:09:46.180734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:09:46.180743 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:09:46.180791 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:09:46.181380 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.181861 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.182667 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.182875 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.182959 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:09:46.184391 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:09:46.184393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:09:46.184393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:09:46.184401 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:09:46.186291 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:09:46.186709 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:09:46.186710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:09:46.186710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:09:46.186769 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:09:46.186983 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:09:46.186985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:09:46.186985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:09:46.187121 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:09:46.187221 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 10:09:46.187435 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 10:09:46.187539 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 10:09:46.189778 25737 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:09:46.191500 25737 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:09:46.202487 26233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:09:46.195079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:38151 (local address 127.0.0.1:49690)
0925 10:09:46.195251 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 10:09:46.195293 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:09:46.195356 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:09:46.197771 (+ 2415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:09:46.197775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:09:46.197778 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:09:46.197866 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:09:46.197869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.199487 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.199488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:09:46.200835 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:09:46.200838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.202157 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.202158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:09:46.202250 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:09:46.202269 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 10:09:46.202315 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 10:09:46.202373 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":112,"threads_started":1}
I20260925 10:09:46.203413 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:09:46.195812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49690 (local address 127.14.41.66:38151)
0925 10:09:46.197456 (+ 1644us) server_negotiation.cc:206] Beginning negotiation
0925 10:09:46.197461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:09:46.197472 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:09:46.197635 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:09:46.197637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:09:46.197652 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:09:46.197766 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:09:46.197959 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.199405 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.200959 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.201744 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.203072 (+ 1328us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:09:46.203173 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 10:09:46.203202 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 10:09:46.203301 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1604,"thread_start_us":75,"threads_started":1}
I20260925 10:09:46.204339 26021 tablet_service.cc:1467] Processing CreateTablet for tablet 21346ce38c8948d986f9e177407294f4 (DEFAULT_TABLE table=test-table [id=deb1223af91e4f26846121965f87a6f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:09:46.204742 26021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21346ce38c8948d986f9e177407294f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:09:46.205823 26234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:09:46.195510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:38207 (local address 127.0.0.1:56722)
0925 10:09:46.198009 (+ 2499us) negotiation.cc:107] Waiting for socket to connect
0925 10:09:46.198020 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:09:46.198054 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:09:46.198548 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:09:46.198549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:09:46.198551 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:09:46.198597 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:09:46.198598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.201269 (+ 2671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.201270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:09:46.202351 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:09:46.202353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.205589 (+ 3236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.205591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:09:46.205658 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:09:46.205664 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:09:46.205696 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 10:09:46.205739 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2484,"thread_start_us":64,"threads_started":1}
I20260925 10:09:46.206207 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:09:46.195618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.14.41.65:38207)
0925 10:09:46.198385 (+ 2767us) server_negotiation.cc:206] Beginning negotiation
0925 10:09:46.198389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:09:46.198397 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:09:46.198427 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:09:46.198429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:09:46.198447 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:09:46.198664 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:09:46.198733 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.199756 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.202503 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.203290 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.205939 (+ 2649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:09:46.206034 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:09:46.206065 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 10:09:46.206118 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2733,"thread_start_us":59,"threads_started":1}
I20260925 10:09:46.206614 26235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:09:46.195340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:39665 (local address 127.0.0.1:57426)
0925 10:09:46.197879 (+ 2539us) negotiation.cc:107] Waiting for socket to connect
0925 10:09:46.197890 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:09:46.197931 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:09:46.199875 (+ 1944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:09:46.199876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:09:46.199878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:09:46.199970 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:09:46.199972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.203393 (+ 3421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.203394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:09:46.204552 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:09:46.204554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.206366 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.206367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:09:46.206446 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:09:46.206451 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:09:46.206482 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 10:09:46.206540 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2518,"thread_start_us":65,"threads_started":1}
I20260925 10:09:46.206995 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:09:46.195792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57426 (local address 127.14.41.67:39665)
0925 10:09:46.198176 (+ 2384us) server_negotiation.cc:206] Beginning negotiation
0925 10:09:46.198180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:09:46.198190 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:09:46.198237 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:09:46.198239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:09:46.198256 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:09:46.200054 (+ 1798us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:09:46.200135 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.201168 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.204680 (+ 3512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.205457 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.206704 (+ 1247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:09:46.206799 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:09:46.206844 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 10:09:46.206890 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2342,"thread_start_us":80,"threads_started":1}
I20260925 10:09:46.208978 26239 tablet_bootstrap.cc:492] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9: Bootstrap starting.
I20260925 10:09:46.209000 26148 tablet_service.cc:1467] Processing CreateTablet for tablet 21346ce38c8948d986f9e177407294f4 (DEFAULT_TABLE table=test-table [id=deb1223af91e4f26846121965f87a6f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:09:46.209353 26148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21346ce38c8948d986f9e177407294f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:09:46.210042 26239 tablet_bootstrap.cc:654] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9: Neither blocks nor log segments found. Creating new log.
I20260925 10:09:46.210350 26239 log.cc:824] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:09:46.211149 26239 tablet_bootstrap.cc:492] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9: No bootstrap required, opened a new log
I20260925 10:09:46.211213 26239 ts_tablet_manager.cc:1397] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:09:46.212447 25883 tablet_service.cc:1467] Processing CreateTablet for tablet 21346ce38c8948d986f9e177407294f4 (DEFAULT_TABLE table=test-table [id=deb1223af91e4f26846121965f87a6f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:09:46.212867 25883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21346ce38c8948d986f9e177407294f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:09:46.213367 26239 raft_consensus.cc:348] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ddc0c78407d40ba9d7c967fc71e99c9" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38151 } } peers { permanent_uuid: "93ded2dc783f4d56b64278476e69f1a5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39665 } } peers { permanent_uuid: "db95a542652546ac83c760b59ef84def" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 38207 } }
I20260925 10:09:46.213529 26239 raft_consensus.cc:374] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:09:46.213562 26239 raft_consensus.cc:729] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ddc0c78407d40ba9d7c967fc71e99c9, State: Initialized, Role: FOLLOWER
I20260925 10:09:46.213680 26239 consensus_queue.cc:260] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ddc0c78407d40ba9d7c967fc71e99c9" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38151 } } peers { permanent_uuid: "93ded2dc783f4d56b64278476e69f1a5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39665 } } peers { permanent_uuid: "db95a542652546ac83c760b59ef84def" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 38207 } }
I20260925 10:09:46.213953 26239 ts_tablet_manager.cc:1428] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 10:09:46.214527 26087 heartbeater.cc:503] Master 127.14.41.126:34621 was elected leader, sending a full tablet report...
I20260925 10:09:46.216622 26242 tablet_bootstrap.cc:492] T 21346ce38c8948d986f9e177407294f4 P db95a542652546ac83c760b59ef84def: Bootstrap starting.
I20260925 10:09:46.217346 26242 tablet_bootstrap.cc:654] T 21346ce38c8948d986f9e177407294f4 P db95a542652546ac83c760b59ef84def: Neither blocks nor log segments found. Creating new log.
I20260925 10:09:46.217785 26242 log.cc:824] T 21346ce38c8948d986f9e177407294f4 P db95a542652546ac83c760b59ef84def: Log is configured to *not* fsync() on all Append() calls
I20260925 10:09:46.218293 26244 tablet_bootstrap.cc:492] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5: Bootstrap starting.
I20260925 10:09:46.219264 26244 tablet_bootstrap.cc:654] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5: Neither blocks nor log segments found. Creating new log.
I20260925 10:09:46.219610 26244 log.cc:824] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5: Log is configured to *not* fsync() on all Append() calls
I20260925 10:09:46.220501 26244 tablet_bootstrap.cc:492] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5: No bootstrap required, opened a new log
I20260925 10:09:46.220625 26244 ts_tablet_manager.cc:1397] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:09:46.222450 26244 raft_consensus.cc:348] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ddc0c78407d40ba9d7c967fc71e99c9" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38151 } } peers { permanent_uuid: "93ded2dc783f4d56b64278476e69f1a5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39665 } } peers { permanent_uuid: "db95a542652546ac83c760b59ef84def" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 38207 } }
I20260925 10:09:46.223078 26242 tablet_bootstrap.cc:492] T 21346ce38c8948d986f9e177407294f4 P db95a542652546ac83c760b59ef84def: No bootstrap required, opened a new log
I20260925 10:09:46.223170 26242 ts_tablet_manager.cc:1397] T 21346ce38c8948d986f9e177407294f4 P db95a542652546ac83c760b59ef84def: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260925 10:09:46.225818 26242 raft_consensus.cc:348] T 21346ce38c8948d986f9e177407294f4 P db95a542652546ac83c760b59ef84def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ddc0c78407d40ba9d7c967fc71e99c9" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38151 } } peers { permanent_uuid: "93ded2dc783f4d56b64278476e69f1a5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39665 } } peers { permanent_uuid: "db95a542652546ac83c760b59ef84def" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 38207 } }
I20260925 10:09:46.226070 26242 raft_consensus.cc:374] T 21346ce38c8948d986f9e177407294f4 P db95a542652546ac83c760b59ef84def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:09:46.226168 26242 raft_consensus.cc:729] T 21346ce38c8948d986f9e177407294f4 P db95a542652546ac83c760b59ef84def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db95a542652546ac83c760b59ef84def, State: Initialized, Role: FOLLOWER
I20260925 10:09:46.226362 26242 consensus_queue.cc:260] T 21346ce38c8948d986f9e177407294f4 P db95a542652546ac83c760b59ef84def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ddc0c78407d40ba9d7c967fc71e99c9" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38151 } } peers { permanent_uuid: "93ded2dc783f4d56b64278476e69f1a5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39665 } } peers { permanent_uuid: "db95a542652546ac83c760b59ef84def" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 38207 } }
I20260925 10:09:46.226696 25949 heartbeater.cc:503] Master 127.14.41.126:34621 was elected leader, sending a full tablet report...
I20260925 10:09:46.226693 26242 ts_tablet_manager.cc:1428] T 21346ce38c8948d986f9e177407294f4 P db95a542652546ac83c760b59ef84def: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 10:09:46.226818 26244 raft_consensus.cc:374] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:09:46.226876 26244 raft_consensus.cc:729] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93ded2dc783f4d56b64278476e69f1a5, State: Initialized, Role: FOLLOWER
I20260925 10:09:46.227008 26244 consensus_queue.cc:260] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ddc0c78407d40ba9d7c967fc71e99c9" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38151 } } peers { permanent_uuid: "93ded2dc783f4d56b64278476e69f1a5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39665 } } peers { permanent_uuid: "db95a542652546ac83c760b59ef84def" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 38207 } }
I20260925 10:09:46.227298 26244 ts_tablet_manager.cc:1428] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260925 10:09:46.227452 26221 heartbeater.cc:503] Master 127.14.41.126:34621 was elected leader, sending a full tablet report...
Sep 25 10:09:46 dist-test-slave-s6tm krb5kdc[25261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330985, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for HTTP/127.14.41.84@KRBTEST.COM
W20260925 10:09:46.315171 25950 tablet.cc:2369] T 21346ce38c8948d986f9e177407294f4 P db95a542652546ac83c760b59ef84def: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:09:46.329839 26241 raft_consensus.cc:482] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:09:46.330010 26241 raft_consensus.cc:504] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ddc0c78407d40ba9d7c967fc71e99c9" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38151 } } peers { permanent_uuid: "93ded2dc783f4d56b64278476e69f1a5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39665 } } peers { permanent_uuid: "db95a542652546ac83c760b59ef84def" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 38207 } }
I20260925 10:09:46.330426 26241 leader_election.cc:288] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93ded2dc783f4d56b64278476e69f1a5 (127.14.41.67:39665), db95a542652546ac83c760b59ef84def (127.14.41.65:38207)
I20260925 10:09:46.334461 26089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:09:46.330606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:38207 (local address 127.14.41.66:45213)
0925 10:09:46.330705 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 10:09:46.330716 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:09:46.330754 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:09:46.331095 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:09:46.331096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:09:46.331098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:09:46.331172 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:09:46.331174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.332113 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.332114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:09:46.333429 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:09:46.333433 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.334170 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.334171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:09:46.334255 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:09:46.334260 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:09:46.334328 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 10:09:46.334375 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 10:09:46.334475 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:09:46.330828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:45213 (local address 127.14.41.65:38207)
0925 10:09:46.330930 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 10:09:46.330932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:09:46.330940 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:09:46.330968 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:09:46.330969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:09:46.330978 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:09:46.331022 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:09:46.331249 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.332019 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.333524 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.334074 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.334147 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:09:46.334155 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:09:46.334344 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 10:09:46.334404 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 10:09:46.334898 25884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21346ce38c8948d986f9e177407294f4" candidate_uuid: "1ddc0c78407d40ba9d7c967fc71e99c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db95a542652546ac83c760b59ef84def" is_pre_election: true
I20260925 10:09:46.335105 25884 raft_consensus.cc:2454] T 21346ce38c8948d986f9e177407294f4 P db95a542652546ac83c760b59ef84def [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ddc0c78407d40ba9d7c967fc71e99c9 in term 0.
I20260925 10:09:46.335378 25976 leader_election.cc:302] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ddc0c78407d40ba9d7c967fc71e99c9, db95a542652546ac83c760b59ef84def; no voters:
I20260925 10:09:46.335533 26241 raft_consensus.cc:2781] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:09:46.335614 26241 raft_consensus.cc:482] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:09:46.335645 26241 raft_consensus.cc:3037] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:09:46.336649 26241 raft_consensus.cc:504] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ddc0c78407d40ba9d7c967fc71e99c9" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38151 } } peers { permanent_uuid: "93ded2dc783f4d56b64278476e69f1a5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39665 } } peers { permanent_uuid: "db95a542652546ac83c760b59ef84def" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 38207 } }
I20260925 10:09:46.336786 26241 leader_election.cc:288] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [CANDIDATE]: Term 1 election: Requested vote from peers 93ded2dc783f4d56b64278476e69f1a5 (127.14.41.67:39665), db95a542652546ac83c760b59ef84def (127.14.41.65:38207)
I20260925 10:09:46.337218 25884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21346ce38c8948d986f9e177407294f4" candidate_uuid: "1ddc0c78407d40ba9d7c967fc71e99c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db95a542652546ac83c760b59ef84def"
I20260925 10:09:46.337296 25884 raft_consensus.cc:3037] T 21346ce38c8948d986f9e177407294f4 P db95a542652546ac83c760b59ef84def [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:09:46.338186 25884 raft_consensus.cc:2454] T 21346ce38c8948d986f9e177407294f4 P db95a542652546ac83c760b59ef84def [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ddc0c78407d40ba9d7c967fc71e99c9 in term 1.
I20260925 10:09:46.338395 25976 leader_election.cc:302] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ddc0c78407d40ba9d7c967fc71e99c9, db95a542652546ac83c760b59ef84def; no voters:
I20260925 10:09:46.338522 26241 raft_consensus.cc:2781] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:09:46.338656 26241 raft_consensus.cc:686] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [term 1 LEADER]: Becoming Leader. State: Replica: 1ddc0c78407d40ba9d7c967fc71e99c9, State: Running, Role: LEADER
I20260925 10:09:46.338748 26241 consensus_queue.cc:237] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ddc0c78407d40ba9d7c967fc71e99c9" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38151 } } peers { permanent_uuid: "93ded2dc783f4d56b64278476e69f1a5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39665 } } peers { permanent_uuid: "db95a542652546ac83c760b59ef84def" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 38207 } }
I20260925 10:09:46.339720 25737 catalog_manager.cc:5495] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ddc0c78407d40ba9d7c967fc71e99c9 (127.14.41.66). New cstate: current_term: 1 leader_uuid: "1ddc0c78407d40ba9d7c967fc71e99c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ddc0c78407d40ba9d7c967fc71e99c9" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93ded2dc783f4d56b64278476e69f1a5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db95a542652546ac83c760b59ef84def" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20260925 10:09:46.345611 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:09:46.330886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:42191 (local address 127.14.41.67:39665)
0925 10:09:46.331101 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 10:09:46.331103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:09:46.331111 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:09:46.331139 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:09:46.331140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:09:46.331148 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:09:46.331196 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:09:46.336484 (+ 5288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.337116 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.344340 (+ 7224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.344961 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.345024 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:09:46.345032 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:09:46.345487 (+ 455us) server_negotiation.cc:299] Negotiation successful
0925 10:09:46.345538 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260925 10:09:46.345840 26248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:09:46.330607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:39665 (local address 127.14.41.66:42191)
0925 10:09:46.330898 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 10:09:46.330909 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:09:46.330961 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:09:46.336262 (+ 5301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:09:46.336264 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:09:46.336283 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:09:46.336370 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:09:46.336371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.337236 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.337236 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:09:46.338213 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:09:46.338216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.345267 (+ 7051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.345269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:09:46.345393 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:09:46.345398 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:09:46.345707 (+ 309us) client_negotiation.cc:240] Negotiation successful
0925 10:09:46.345754 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":16,"thread_start_us":177,"threads_started":1}
I20260925 10:09:46.346276 26175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21346ce38c8948d986f9e177407294f4" candidate_uuid: "1ddc0c78407d40ba9d7c967fc71e99c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ded2dc783f4d56b64278476e69f1a5" is_pre_election: true
I20260925 10:09:46.346441 26175 raft_consensus.cc:2454] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ddc0c78407d40ba9d7c967fc71e99c9 in term 0.
I20260925 10:09:46.346545 26174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21346ce38c8948d986f9e177407294f4" candidate_uuid: "1ddc0c78407d40ba9d7c967fc71e99c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ded2dc783f4d56b64278476e69f1a5"
I20260925 10:09:46.346590 26174 raft_consensus.cc:3037] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:09:46.347512 26174 raft_consensus.cc:2454] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ddc0c78407d40ba9d7c967fc71e99c9 in term 1.
I20260925 10:09:46.350762 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:09:46.344802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49704 (local address 127.14.41.66:38151)
0925 10:09:46.344869 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 10:09:46.344871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:09:46.345132 (+ 261us) server_negotiation.cc:407] Connection header received
0925 10:09:46.345212 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:09:46.345213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:09:46.345238 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:09:46.345305 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:09:46.345489 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.346203 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.347377 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:09:46.347705 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:09:46.347770 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:09:46.348008 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:09:46.348171 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:09:46.348228 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 10:09:46.350657 (+ 2429us) server_negotiation.cc:299] Negotiation successful
0925 10:09:46.350682 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:09:46.356405 25884 raft_consensus.cc:1261] T 21346ce38c8948d986f9e177407294f4 P db95a542652546ac83c760b59ef84def [term 1 FOLLOWER]: Refusing update from remote peer 1ddc0c78407d40ba9d7c967fc71e99c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:09:46.356804 26241 consensus_queue.cc:1035] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db95a542652546ac83c760b59ef84def" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:09:46.356922 26174 raft_consensus.cc:1261] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5 [term 1 FOLLOWER]: Refusing update from remote peer 1ddc0c78407d40ba9d7c967fc71e99c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:09:46.357205 26241 consensus_queue.cc:1035] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93ded2dc783f4d56b64278476e69f1a5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:09:46.361182 26253 mvcc.cc:205] Tried to move back new op lower bound from 7333195720111562752 to 7333195720044257280. Current Snapshot: MvccSnapshot[applied={T|T < 7333195720111562752}]
I20260925 10:09:46.361967 26252 mvcc.cc:205] Tried to move back new op lower bound from 7333195720111562752 to 7333195720044257280. Current Snapshot: MvccSnapshot[applied={T|T < 7333195720111562752}]
I20260925 10:09:46.363139 26254 mvcc.cc:205] Tried to move back new op lower bound from 7333195720111562752 to 7333195720044257280. Current Snapshot: MvccSnapshot[applied={T|T < 7333195720111562752 or (T in {7333195720111562752})}]
I20260925 10:09:46.365054 25737 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:09:46.365154 25737 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:09:46.365935 25737 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c352fe0c3d64c47b0bedd936db68b97: Sending DeleteTablet for 3 replicas of tablet 21346ce38c8948d986f9e177407294f4
I20260925 10:09:46.366278 25883 tablet_service.cc:1514] Processing DeleteTablet for tablet 21346ce38c8948d986f9e177407294f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:09:46 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:56722
I20260925 10:09:46.366346 26021 tablet_service.cc:1514] Processing DeleteTablet for tablet 21346ce38c8948d986f9e177407294f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:09:46 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:49690
I20260925 10:09:46.366578 26148 tablet_service.cc:1514] Processing DeleteTablet for tablet 21346ce38c8948d986f9e177407294f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:09:46 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:57426
I20260925 10:09:46.367053 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 25818
I20260925 10:09:46.372084 26263 tablet_replica.cc:330] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9: stopping tablet replica
I20260925 10:09:46.372262 26263 raft_consensus.cc:2229] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:09:46.372538 26263 raft_consensus.cc:2258] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:09:46.373144 26263 ts_tablet_manager.cc:1905] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:09:46.375051 26263 ts_tablet_manager.cc:1918] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:09:46.375216 26263 log.cc:1196] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/21346ce38c8948d986f9e177407294f4
I20260925 10:09:46.375574 26263 ts_tablet_manager.cc:1939] T 21346ce38c8948d986f9e177407294f4 P 1ddc0c78407d40ba9d7c967fc71e99c9: Deleting consensus metadata
I20260925 10:09:46.376049 26264 tablet_replica.cc:330] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5: stopping tablet replica
I20260925 10:09:46.376148 25722 catalog_manager.cc:4841] TS 1ddc0c78407d40ba9d7c967fc71e99c9 (127.14.41.66:38151): tablet 21346ce38c8948d986f9e177407294f4 (table test-table [id=deb1223af91e4f26846121965f87a6f7]) successfully deleted
I20260925 10:09:46.376260 26264 raft_consensus.cc:2229] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:09:46.376399 26264 raft_consensus.cc:2258] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:09:46.376886 26264 ts_tablet_manager.cc:1905] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:09:46.378592 26264 ts_tablet_manager.cc:1918] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:09:46.378727 26264 log.cc:1196] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/21346ce38c8948d986f9e177407294f4
I20260925 10:09:46.379375 26264 ts_tablet_manager.cc:1939] T 21346ce38c8948d986f9e177407294f4 P 93ded2dc783f4d56b64278476e69f1a5: Deleting consensus metadata
I20260925 10:09:46.379896 25722 catalog_manager.cc:4841] TS 93ded2dc783f4d56b64278476e69f1a5 (127.14.41.67:39665): tablet 21346ce38c8948d986f9e177407294f4 (table test-table [id=deb1223af91e4f26846121965f87a6f7]) successfully deleted
W20260925 10:09:46.384275 25724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.41.65:38207 (error 108)
I20260925 10:09:46.385232 26235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:09:46.385110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:38207 (local address 127.0.0.1:56724)
0925 10:09:46.385162 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 10:09:46.385195 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.65:38207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 10:09:46.385475 25724 catalog_manager.cc:4584] TS db95a542652546ac83c760b59ef84def (127.14.41.65:38207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 21346ce38c8948d986f9e177407294f4: Network error: Client connection negotiation failed: client connection to 127.14.41.65:38207: connect: Connection refused (error 111)
I20260925 10:09:46.385771 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 25955
I20260925 10:09:46.391374 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 26092
I20260925 10:09:46.396353 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 25705
I20260925 10:09:46.402113 14501 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 10:09:46.747097 14501 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 10:09:46.747267 14501 mini_ranger.cc:66] Stopping Ranger...
I20260925 10:09:47.092391 14501 mini_ranger.cc:68] Stopped Ranger
2026-09-25 10:09:47.092 UTC [25291] LOG: received smart shutdown request
2026-09-25 10:09:47.096 UTC [25291] LOG: background worker "logical replication launcher" (PID 25300) exited with exit code 1
2026-09-25 10:09:47.096 UTC [25295] LOG: shutting down
2026-09-25 10:09:47.286 UTC [25291] LOG: database system is shut down
2026-09-25T10:09:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48086 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:09:47 dist-test-slave-s6tm krb5kdc[26274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:09:47 dist-test-slave-s6tm krb5kdc[26274](info): set up 2 sockets
Sep 25 10:09:47 dist-test-slave-s6tm krb5kdc[26274](info): commencing operation
krb5kdc: starting...
W20260925 10:09:48.469426 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:09:48 dist-test-slave-s6tm krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790330988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:09:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:09:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.14.41.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.14.41.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.14.41.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/rangeradmin_127.14.41.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.14.41.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/rangeradmin_127.14.41.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.14.41.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.14.41.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.14.41.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/rangerlookup_127.14.41.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.14.41.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/rangerlookup_127.14.41.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.41.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.14.41.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.41.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/HTTP_127.14.41.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.41.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/HTTP_127.14.41.84@KRBTEST.COM.keytab.
I20260925 10:09:48.644286 14501 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskdDYjLr/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/postgres -l logfile start
2026-09-25 10:09:51.348 UTC [26304] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 10:09:51.348 UTC [26304] LOG: listening on IPv4 address "127.14.41.84", port 49797
2026-09-25 10:09:51.353 UTC [26304] LOG: listening on Unix socket "/tmp/.s.PGSQL.49797"
2026-09-25 10:09:51.391 UTC [26307] LOG: database system was shut down at 2026-09-25 10:09:49 UTC
2026-09-25 10:09:51.395 UTC [26304] LOG: database system is ready to accept connections
I20260925 10:09:52.353247 14501 mini_postgres.cc:96] Postgres bound to 49797
2026-09-25 10:09:52.360 UTC [26315] FATAL: database "slave" does not exist
127.14.41.84:49797 - accepting connections
I20260925 10:09:52.360735 14501 mini_ranger.cc:161] Starting Ranger...
I20260925 10:09:52.378098 14501 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 10:09:52.931962 14501 mini_ranger.cc:87] Created ranger Postgres database
I20260925 10:09:52.932053 14501 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ranger-admin
2026-09-25 10:09:53,226 [I] DB FLAVOR :POSTGRES
2026-09-25 10:09:53,226 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:09:53,226 [I] Checking connection..
2026-09-25 10:09:53,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 10:09:53,497 [I] Checking connection passed.
2026-09-25 10:09:53,498 [I] --------- Verifying version history table ---------
2026-09-25 10:09:53,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:09:53,768 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:09:53,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:09:54,007 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:09:54,007 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 10:09:54,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 10:09:54.275 UTC [26379] WARNING: there is no transaction in progress
2026-09-25 10:09:54,283 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 10:09:54,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:09:54,522 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 10:09:54,523 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:09:54,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 10:09:54,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 10:09:55,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-s6tm.c.gcp-upstream.internal','N') ;"
2026-09-25 10:09:55,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:09:55,603 [I] Table x_portal_user does not exist in database ranger
2026-09-25 10:09:55,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:09:55,853 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 10:09:55,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:09:56,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:09:56,360 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 10:09:56,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 10:09:57.418 UTC [26487] WARNING: there is no transaction in progress
2026-09-25 10:09:57.436 UTC [26487] WARNING: there is no transaction in progress
2026-09-25 10:09:57.454 UTC [26487] WARNING: there is no transaction in progress
2026-09-25 10:09:57.471 UTC [26487] WARNING: there is no transaction in progress
2026-09-25 10:09:57.489 UTC [26487] WARNING: there is no transaction in progress
2026-09-25 10:09:57.510 UTC [26487] WARNING: there is no transaction in progress
2026-09-25 10:09:57.568 UTC [26487] WARNING: there is no transaction in progress
2026-09-25 10:09:57.574 UTC [26487] WARNING: there is no transaction in progress
2026-09-25 10:09:57.589 UTC [26487] WARNING: there is no transaction in progress
2026-09-25 10:09:57.600 UTC [26487] WARNING: there is no transaction in progress
2026-09-25 10:09:57.611 UTC [26487] WARNING: there is no transaction in progress
2026-09-25 10:09:57.627 UTC [26487] WARNING: there is no transaction in progress
2026-09-25 10:09:57.638 UTC [26487] WARNING: there is no transaction in progress
2026-09-25 10:09:57.645 UTC [26487] WARNING: there is no transaction in progress
2026-09-25 10:09:57.652 UTC [26487] WARNING: there is no transaction in progress
2026-09-25 10:09:57.659 UTC [26487] WARNING: there is no transaction in progress
2026-09-25 10:09:58,406 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 10:09:58,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 10:09:58,652 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 10:09:58,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:09:58,908 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 10:09:58,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:09:59,148 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 10:09:59,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:09:59,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:09:59,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-s6tm.c.gcp-upstream.internal';"
2026-09-25 10:09:59,930 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 10:09:59,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 10:10:00,190 [I] DB_PATCHES have already been applied
I20260925 10:10:00.195899 14501 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ranger-admin:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskdDYjLr/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 10:10:00.196040 14501 mini_ranger.cc:193] Using host: 127.14.41.84
I20260925 10:10:00.199225 14501 mini_ranger.cc:227] Ranger admin URL: http://127.14.41.84:50751
Sep 25, 2026 10:10:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:10:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:10:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:10:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:10:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:10:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:10:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 10:10:01.318306 14501 mini_ranger.cc:160] Time spent starting Ranger: real 8.958s user 0.000s sys 0.010s
Sep 25, 2026 10:10:20 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:10:20 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 10:10:20 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:10:20 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:10:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:10:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:10:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:10:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:10:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:10:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:10:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:10:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:10:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:10:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:10:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:10:20 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:10:20 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:10:21 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 10:10:22.396456 14501 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.14.41.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.14.41.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.14.41.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/rangerkms_127.14.41.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.14.41.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/rangerkms_127.14.41.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.14.41.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.14.41.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.14.41.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/HTTP_127.14.41.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.14.41.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/HTTP_127.14.41.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 10:10:22 dist-test-slave-s6tm krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331022, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 10:10:22.482867 14501 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 10:10:22.492352 14501 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 10:10:23.488878 14501 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 10:10:23.488972 14501 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ranger-kms
I20260925 10:10:23.488991 14501 mini_ranger_kms.cc:227] Using postgres at 127.14.41.84:49797
2026-09-25 10:10:23,666 [I] DB FLAVOR :POSTGRES
2026-09-25 10:10:23,666 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:10:23,666 [I] Checking connection
2026-09-25 10:10:23,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 10:10:23,911 [I] connection success
2026-09-25 10:10:23,911 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 10:10:23,911 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 10:10:23,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 10:10:24,147 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 10:10:24,147 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:10:24,147 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 10:10:24,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.14.41.84:49797/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 10:10:24,412 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 10:10:24.873041 14501 mini_ranger_kms.cc:310] Created kms service
I20260925 10:10:25.055605 14501 mini_ranger_kms.cc:326] Created kudu user
I20260925 10:10:25.105464 14501 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 10:10:25 dist-test-slave-s6tm krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331022, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.14.41.84@KRBTEST.COM
I20260925 10:10:25.524205 14501 mini_ranger_kms.cc:384] Added ranger policy
I20260925 10:10:25.524407 14501 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ranger-kms:/tmp/dist-test-taskdDYjLr/build/release/bin/postgresql.jar:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskdDYjLr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 10:10:25.524430 14501 mini_ranger_kms.cc:240] Using host: 127.14.41.84
I20260925 10:10:25.527698 14501 mini_ranger_kms.cc:276] Ranger KMS PID: 26667
I20260925 10:10:25.527786 14501 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.14.41.84:32929
Sep 25, 2026 10:10:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:10:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:10:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:10:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:10:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:10:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 10:10:27.825484 14501 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.343s user 0.000s sys 0.159s
I20260925 10:10:27.825582 14501 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 10:10:27.825630 14501 mini_ranger_kms.cc:401] 127.14.41.84:32929/kms/v1/keys
Sep 25 10:10:28 dist-test-slave-s6tm krb5kdc[26274](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790331028, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.41.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 10:10:28 dist-test-slave-s6tm krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331028, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.41.84@KRBTEST.COM for HTTP/127.14.41.84@KRBTEST.COM
Sep 25 10:10:28 dist-test-slave-s6tm krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331028, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.14.41.84@KRBTEST.COM for HTTP/127.14.41.84@KRBTEST.COM
Sep 25 10:10:31 dist-test-slave-s6tm krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:10:31.107741 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:37941
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:37941
--ranger_config_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.41.84:32929/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:10:31.191476 26718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:31.191705 26718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:31.191728 26718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:31.193051 26718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:10:31.193100 26718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:31.193120 26718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:31.193135 26718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:10:31.193146 26718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:10:31.194811 26718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/wal
--ranger_kms_url=127.14.41.84:32929/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:37941
--ranger_config_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:37941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:31.195096 26718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:31.195355 26718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:10:31.201181 26718 server_base.cc:1034] running on GCE node
W20260925 10:10:31.201151 26724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:31.201156 26726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:31.201156 26723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:31.201745 26718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:31.202045 26718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:31.203214 26718 hybrid_clock.cc:648] HybridClock initialized: now 1790331031203200 us; error 46 us; skew 500 ppm
Sep 25 10:10:31 dist-test-slave-s6tm krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331031, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:31.210775 26718 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:10:31.211279 26718 webserver.cc:460] Webserver started at http://127.14.41.126:33125/ using document root <none> and password file <none>
I20260925 10:10:31.211539 26718 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:31.211604 26718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:31.211769 26718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:10:31 dist-test-slave-s6tm krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331031, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for HTTP/127.14.41.84@KRBTEST.COM
I20260925 10:10:31.259820 26718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "f9b002b10bcd4e62bc7c51a7f2b2fde0"
format_stamp: "Formatted at 2026-09-25 10:10:31 on dist-test-slave-s6tm"
server_key: "35dd4e3312bb55f57f7d2e526579e28e"
server_key_iv: "72cb3f52fa73f0a23966f939f189a771"
server_key_version: "kuduclusterkey@0"
I20260925 10:10:31.260349 26718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "f9b002b10bcd4e62bc7c51a7f2b2fde0"
format_stamp: "Formatted at 2026-09-25 10:10:31 on dist-test-slave-s6tm"
server_key: "35dd4e3312bb55f57f7d2e526579e28e"
server_key_iv: "72cb3f52fa73f0a23966f939f189a771"
server_key_version: "kuduclusterkey@0"
I20260925 10:10:31.262214 26718 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:10:31.274451 26733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:31.274715 26718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:10:31.274829 26718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "f9b002b10bcd4e62bc7c51a7f2b2fde0"
format_stamp: "Formatted at 2026-09-25 10:10:31 on dist-test-slave-s6tm"
server_key: "35dd4e3312bb55f57f7d2e526579e28e"
server_key_iv: "72cb3f52fa73f0a23966f939f189a771"
server_key_version: "kuduclusterkey@0"
I20260925 10:10:31.274950 26718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:31.284940 26718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:31.287189 26718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:31.287369 26718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:31.292407 26718 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:37941
I20260925 10:10:31.292493 26785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:37941 every 8 connection(s)
I20260925 10:10:31.292878 26718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:10:31.293593 26786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:10:31.294239 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 26718
I20260925 10:10:31.294341 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:10:31.294582 14501 external_mini_cluster.cc:1428] Setting key 1ff7641938917fdf555704784f53c8a4
I20260925 10:10:31.297013 26786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0: Bootstrap starting.
I20260925 10:10:31.297808 26786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0: Neither blocks nor log segments found. Creating new log.
I20260925 10:10:31.298107 26786 log.cc:824] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0: Log is configured to *not* fsync() on all Append() calls
I20260925 10:10:31.298894 26786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0: No bootstrap required, opened a new log
I20260925 10:10:31.301334 26786 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b002b10bcd4e62bc7c51a7f2b2fde0" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37941 } }
I20260925 10:10:31.301512 26786 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:10:31.301587 26786 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b002b10bcd4e62bc7c51a7f2b2fde0, State: Initialized, Role: FOLLOWER
I20260925 10:10:31.301759 26786 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b002b10bcd4e62bc7c51a7f2b2fde0" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37941 } }
I20260925 10:10:31.301859 26786 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:10:31.301916 26786 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:10:31.301975 26786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:10:31.302621 26786 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b002b10bcd4e62bc7c51a7f2b2fde0" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37941 } }
I20260925 10:10:31.302738 26786 leader_election.cc:302] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9b002b10bcd4e62bc7c51a7f2b2fde0; no voters:
I20260925 10:10:31.302963 26786 leader_election.cc:288] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [CANDIDATE]: Term 1 election: Requested vote from peers
Sep 25 10:10:31 dist-test-slave-s6tm krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:10:31.303025 26791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:10:31.303421 26791 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [term 1 LEADER]: Becoming Leader. State: Replica: f9b002b10bcd4e62bc7c51a7f2b2fde0, State: Running, Role: LEADER
I20260925 10:10:31.303547 26786 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:10:31.303572 26791 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b002b10bcd4e62bc7c51a7f2b2fde0" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37941 } }
I20260925 10:10:31.303980 26793 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9b002b10bcd4e62bc7c51a7f2b2fde0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b002b10bcd4e62bc7c51a7f2b2fde0" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37941 } } }
I20260925 10:10:31.304143 26793 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [sys.catalog]: This master's current role is: LEADER
I20260925 10:10:31.304363 26792 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9b002b10bcd4e62bc7c51a7f2b2fde0. Latest consensus state: current_term: 1 leader_uuid: "f9b002b10bcd4e62bc7c51a7f2b2fde0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b002b10bcd4e62bc7c51a7f2b2fde0" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37941 } } }
I20260925 10:10:31.304498 26792 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [sys.catalog]: This master's current role is: LEADER
I20260925 10:10:31.304678 26795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:10:31.305259 26795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:10:31.305668 26786 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 10:10:31.306638 26807 mvcc.cc:205] Tried to move back new op lower bound from 7333195904230412288 to 7333195904220037120. Current Snapshot: MvccSnapshot[applied={T|T < 7333195904230412288}]
I20260925 10:10:31.310463 26789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:31.295209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.14.41.126:37941)
0925 10:10:31.295411 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:31.295422 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:31.295519 (+ 97us) server_negotiation.cc:407] Connection header received
0925 10:10:31.296233 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:31.296244 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:31.296315 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:31.296471 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:31.297273 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:31.298626 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:31.299749 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:31.300476 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:31.300600 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:31.303735 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:31.303740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:31.303740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:31.303755 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:31.305886 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:31.306354 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:31.306358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:31.306358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:31.306461 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:31.306670 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:31.306671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:31.306671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:31.308631 (+ 1960us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:31.309409 (+ 778us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:31.309530 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 10:10:31.310123 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260925 10:10:31.312675 26795 catalog_manager.cc:1347] Generated new cluster ID: a0391849042746a285bfd1bb68556d69
I20260925 10:10:31.312734 26795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:10:31.341867 26795 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:10:31.342643 26795 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:10:31.347802 26795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0: Generated new TSK 0
I20260925 10:10:31.348022 26795 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
I20260925 10:10:32.035918 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.41.84:32929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37941
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:10:32.153335 26831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:32.153543 26831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:32.153566 26831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:32.154790 26831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:32.154838 26831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:32.154865 26831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:10:32.156486 26831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.14.41.84:32929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:32.156723 26831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:32.156987 26831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:32.163210 26837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:32.163210 26839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:32.163550 26836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:32.163736 26831 server_base.cc:1034] running on GCE node
I20260925 10:10:32.164009 26831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:32.164301 26831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:32.165475 26831 hybrid_clock.cc:648] HybridClock initialized: now 1790331032165461 us; error 32 us; skew 500 ppm
Sep 25 10:10:32 dist-test-slave-s6tm krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331032, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:32.168524 26831 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:10:32.169082 26831 webserver.cc:460] Webserver started at http://127.14.41.65:44705/ using document root <none> and password file <none>
I20260925 10:10:32.169363 26831 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:32.169415 26831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:32.169574 26831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:10:32 dist-test-slave-s6tm krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331032, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for HTTP/127.14.41.84@KRBTEST.COM
I20260925 10:10:32.185817 26831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "a91da2248cd249338c53668a08276a6c"
format_stamp: "Formatted at 2026-09-25 10:10:32 on dist-test-slave-s6tm"
server_key: "d82d9a8add27ad9c721585a6e580aa8c"
server_key_iv: "febde9dea3c43d0b237dd7db85acee7b"
server_key_version: "kuduclusterkey@0"
I20260925 10:10:32.186372 26831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "a91da2248cd249338c53668a08276a6c"
format_stamp: "Formatted at 2026-09-25 10:10:32 on dist-test-slave-s6tm"
server_key: "d82d9a8add27ad9c721585a6e580aa8c"
server_key_iv: "febde9dea3c43d0b237dd7db85acee7b"
server_key_version: "kuduclusterkey@0"
I20260925 10:10:32.188438 26831 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:10:32.202461 26846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:32.202821 26831 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:10:32.203052 26831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "a91da2248cd249338c53668a08276a6c"
format_stamp: "Formatted at 2026-09-25 10:10:32 on dist-test-slave-s6tm"
server_key: "d82d9a8add27ad9c721585a6e580aa8c"
server_key_iv: "febde9dea3c43d0b237dd7db85acee7b"
server_key_version: "kuduclusterkey@0"
I20260925 10:10:32.203272 26831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:32.213928 26831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:32.217942 26831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:32.218111 26831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:32.218343 26831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:10:32.218667 26831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:10:32.218701 26831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:32.218732 26831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:10:32.218751 26831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:32.225626 26831 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:42813
I20260925 10:10:32.225695 26961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:42813 every 8 connection(s)
I20260925 10:10:32.226199 26831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
Sep 25 10:10:32 dist-test-slave-s6tm krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331032, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:10:32.240151 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 26831
I20260925 10:10:32.240267 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:10:32.240509 14501 external_mini_cluster.cc:1428] Setting key f207b0a0f70d87b6583faf8ccfaa80a6
I20260925 10:10:32.241561 26964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:32.229252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:41635 (local address 127.14.41.126:37941)
0925 10:10:32.229412 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:32.229414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:32.229657 (+ 243us) server_negotiation.cc:407] Connection header received
0925 10:10:32.230340 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:32.230342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:32.230360 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:32.230448 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:32.231292 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:32.232148 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:32.233691 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:32.234035 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:32.234076 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:32.237213 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:32.237214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:32.237214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:32.237222 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:32.239131 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:32.239667 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:32.239668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:32.239669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:32.239745 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:32.240062 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:32.240063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:32.240063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:32.240206 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:32.240250 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:32.241351 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0925 10:10:32.241454 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260925 10:10:32.242331 26965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:32.229235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:37941 (local address 127.14.41.65:41635)
0925 10:10:32.229490 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:32.229548 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:32.230212 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:32.230534 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:32.230538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:32.230827 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:32.231166 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:32.231173 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:32.232265 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:32.232266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:32.233590 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:32.233593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:32.234199 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:32.234200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:32.234353 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:32.235034 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:32.235045 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:32.237089 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:32.239246 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:32.239254 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:32.239256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:32.239575 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:32.239820 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:32.239820 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:32.239821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:32.239983 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:32.240396 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:32.240403 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:32.240638 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 10:10:32.241901 (+ 1263us) client_negotiation.cc:240] Negotiation successful
0925 10:10:32.242084 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":106,"threads_started":1}
I20260925 10:10:32.243072 26962 heartbeater.cc:347] Connected to a master server at 127.14.41.126:37941
I20260925 10:10:32.243175 26962 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:32.243424 26962 heartbeater.cc:511] Master 127.14.41.126:37941 requested a full tablet report, sending...
I20260925 10:10:32.244046 26750 ts_manager.cc:194] Registered new tserver with Master: a91da2248cd249338c53668a08276a6c (127.14.41.65:42813)
I20260925 10:10:32.244964 26750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:41635
WARNING: no policy specified for kudu/127.14.41.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
I20260925 10:10:32.276273 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.41.84:32929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37941
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:10:32.401239 26968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:32.401613 26968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:32.401717 26968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:32.403739 26968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:32.403802 26968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:32.403851 26968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
Sep 25 10:10:32 dist-test-slave-s6tm krb5kdc[26274](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790331032, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:32.406764 26968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.14.41.84:32929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:32.407050 26968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:32.407341 26968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:32.415419 26975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:32.415758 26978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:32.416019 26976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:32.416647 26968 server_base.cc:1034] running on GCE node
I20260925 10:10:32.417059 26968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:32.417341 26968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:32.420598 26968 hybrid_clock.cc:648] HybridClock initialized: now 1790331032420571 us; error 38 us; skew 500 ppm
Sep 25 10:10:32 dist-test-slave-s6tm krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331032, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:32.424186 26968 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:10:32.424729 26968 webserver.cc:460] Webserver started at http://127.14.41.66:40849/ using document root <none> and password file <none>
I20260925 10:10:32.425079 26968 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:32.425202 26968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:32.425384 26968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:10:32 dist-test-slave-s6tm krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331032, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for HTTP/127.14.41.84@KRBTEST.COM
I20260925 10:10:32.440356 26968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "89faffa2bb504b568f49fb1e19fc34ad"
format_stamp: "Formatted at 2026-09-25 10:10:32 on dist-test-slave-s6tm"
server_key: "efa0bed1e6966236559f1a50c6619284"
server_key_iv: "e33063587963000d3a50c0952d7a7a46"
server_key_version: "kuduclusterkey@0"
I20260925 10:10:32.440917 26968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "89faffa2bb504b568f49fb1e19fc34ad"
format_stamp: "Formatted at 2026-09-25 10:10:32 on dist-test-slave-s6tm"
server_key: "efa0bed1e6966236559f1a50c6619284"
server_key_iv: "e33063587963000d3a50c0952d7a7a46"
server_key_version: "kuduclusterkey@0"
I20260925 10:10:32.444782 26968 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 10:10:32.466657 26985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:32.466933 26968 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 10:10:32.467025 26968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "89faffa2bb504b568f49fb1e19fc34ad"
format_stamp: "Formatted at 2026-09-25 10:10:32 on dist-test-slave-s6tm"
server_key: "efa0bed1e6966236559f1a50c6619284"
server_key_iv: "e33063587963000d3a50c0952d7a7a46"
server_key_version: "kuduclusterkey@0"
I20260925 10:10:32.467145 26968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:32.491429 26968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:32.493992 26968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:32.494158 26968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:32.494400 26968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:10:32.494891 26968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:10:32.494944 26968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:32.494976 26968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:10:32.495033 26968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:32.502308 26968 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:41239
I20260925 10:10:32.502430 27099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:41239 every 8 connection(s)
I20260925 10:10:32.502826 26968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:10:32.505043 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 26968
I20260925 10:10:32.505147 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:10:32.505349 14501 external_mini_cluster.cc:1428] Setting key c58a94fbccbc481c7fb5307aec4bb8ae
Sep 25 10:10:32 dist-test-slave-s6tm krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331032, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:10:32.514545 27102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:32.503582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:37941 (local address 127.14.41.66:37487)
0925 10:10:32.503774 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:32.503823 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:32.504543 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:32.504920 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:32.504925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:32.505165 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:32.505437 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:32.505444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:32.506432 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:32.506433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:32.507708 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:32.507711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:32.508352 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:32.508353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:32.508504 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:32.509144 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:32.509154 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:32.510829 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:32.512871 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:32.512878 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:32.512881 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:32.513195 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:32.513408 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:32.513409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:32.513409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:32.513538 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:32.513833 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:32.513839 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:32.514006 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 10:10:32.514126 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 10:10:32.514291 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":114,"threads_started":1}
I20260925 10:10:32.515281 26964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:32.503599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:37487 (local address 127.14.41.126:37941)
0925 10:10:32.503665 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:32.503667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:32.503949 (+ 282us) server_negotiation.cc:407] Connection header received
0925 10:10:32.504722 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:32.504723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:32.504741 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:32.504826 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:32.505551 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:32.506309 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:32.507798 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:32.508197 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:32.508267 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:32.510939 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:32.510940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:32.510941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:32.510948 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:32.512772 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:32.513282 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:32.513283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:32.513283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:32.513343 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:32.513609 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:32.513610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:32.513610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:32.513749 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:32.513808 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:32.515095 (+ 1287us) server_negotiation.cc:299] Negotiation successful
0925 10:10:32.515191 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
WARNING: no policy specified for kudu/127.14.41.67@KRBTEST.COM; defaulting to no policy
I20260925 10:10:32.515676 27100 heartbeater.cc:347] Connected to a master server at 127.14.41.126:37941
I20260925 10:10:32.515771 27100 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:32.515978 27100 heartbeater.cc:511] Master 127.14.41.126:37941 requested a full tablet report, sending...
I20260925 10:10:32.516438 26750 ts_manager.cc:194] Registered new tserver with Master: 89faffa2bb504b568f49fb1e19fc34ad (127.14.41.66:41239)
I20260925 10:10:32.516916 26750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:37487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
I20260925 10:10:32.540371 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.41.84:32929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37941
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:10:32.664577 27105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:32.664883 27105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:32.664928 27105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:32.666924 27105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:32.667013 27105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:32.667063 27105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:10:32.669665 27105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.14.41.84:32929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:37941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:32.670104 27105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:32.670467 27105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:32.677338 27113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:32.677356 27111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:32.678187 27105 server_base.cc:1034] running on GCE node
W20260925 10:10:32.680411 27110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:32.680716 27105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:32.681042 27105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:32.682224 27105 hybrid_clock.cc:648] HybridClock initialized: now 1790331032682200 us; error 38 us; skew 500 ppm
Sep 25 10:10:32 dist-test-slave-s6tm krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331032, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:32.685020 27105 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:10:32.685487 27105 webserver.cc:460] Webserver started at http://127.14.41.67:41075/ using document root <none> and password file <none>
I20260925 10:10:32.685739 27105 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:32.685806 27105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:32.685957 27105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:10:32 dist-test-slave-s6tm krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331032, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for HTTP/127.14.41.84@KRBTEST.COM
I20260925 10:10:32.699364 27105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "b67c9524c1d749cdbce11cd22752eb7e"
format_stamp: "Formatted at 2026-09-25 10:10:32 on dist-test-slave-s6tm"
server_key: "f20e40ed1362b2da1f02f95335ebc85e"
server_key_iv: "c038796cc9c19d6568f56de76d983e87"
server_key_version: "kuduclusterkey@0"
I20260925 10:10:32.699995 27105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "b67c9524c1d749cdbce11cd22752eb7e"
format_stamp: "Formatted at 2026-09-25 10:10:32 on dist-test-slave-s6tm"
server_key: "f20e40ed1362b2da1f02f95335ebc85e"
server_key_iv: "c038796cc9c19d6568f56de76d983e87"
server_key_version: "kuduclusterkey@0"
I20260925 10:10:32.704459 27105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260925 10:10:32.725183 27120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:32.725430 27105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:10:32.725539 27105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "b67c9524c1d749cdbce11cd22752eb7e"
format_stamp: "Formatted at 2026-09-25 10:10:32 on dist-test-slave-s6tm"
server_key: "f20e40ed1362b2da1f02f95335ebc85e"
server_key_iv: "c038796cc9c19d6568f56de76d983e87"
server_key_version: "kuduclusterkey@0"
I20260925 10:10:32.725653 27105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:32.750116 27105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:32.752822 27105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:32.753041 27105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:32.753356 27105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:10:32.753793 27105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:10:32.753890 27105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:32.753961 27105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:10:32.754017 27105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:32.761276 27105 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:42351
I20260925 10:10:32.761484 27233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:42351 every 8 connection(s)
I20260925 10:10:32.761797 27105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
Sep 25 10:10:32 dist-test-slave-s6tm krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331032, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:10:32.769716 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 27105
I20260925 10:10:32.769869 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:10:32.770092 14501 external_mini_cluster.cc:1428] Setting key d8246ac7394898f03528d3791fc1e274
I20260925 10:10:32.772426 26964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:32.762618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:42445 (local address 127.14.41.126:37941)
0925 10:10:32.762694 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:32.762697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:32.762901 (+ 204us) server_negotiation.cc:407] Connection header received
0925 10:10:32.763452 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:32.763453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:32.763466 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:32.763520 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:32.764206 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:32.764793 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:32.765926 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:32.766149 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:32.766238 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:32.768617 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:32.768618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:32.768619 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:32.768628 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:32.770397 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:32.771239 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:32.771241 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:32.771241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:32.771304 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:32.771586 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:32.771586 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:32.771587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:32.771711 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:32.771780 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:32.772221 (+ 441us) server_negotiation.cc:299] Negotiation successful
0925 10:10:32.772314 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:10:32.773218 27236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:32.762610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:37941 (local address 127.14.41.67:42445)
0925 10:10:32.762782 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:32.762826 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:32.763332 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:32.763602 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:32.763606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:32.763803 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:32.764089 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:32.764095 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:32.764912 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:32.764913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:32.765820 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:32.765824 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:32.766261 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:32.766263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:32.766391 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:32.766979 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:32.766988 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:32.768508 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:32.770495 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:32.770503 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:32.770507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:32.771180 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:32.771377 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:32.771378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:32.771378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:32.771507 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:32.771818 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:32.771825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:32.772031 (+ 206us) client_negotiation.cc:769] Sending connection context
0925 10:10:32.772764 (+ 733us) client_negotiation.cc:240] Negotiation successful
0925 10:10:32.772953 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260925 10:10:32.773898 27234 heartbeater.cc:347] Connected to a master server at 127.14.41.126:37941
I20260925 10:10:32.773998 27234 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:32.774204 27234 heartbeater.cc:511] Master 127.14.41.126:37941 requested a full tablet report, sending...
I20260925 10:10:32.774595 26750 ts_manager.cc:194] Registered new tserver with Master: b67c9524c1d749cdbce11cd22752eb7e (127.14.41.67:42351)
I20260925 10:10:32.775014 26750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:42445
I20260925 10:10:32.781971 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:10:32.782588 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 26831
I20260925 10:10:32.788488 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 26968
I20260925 10:10:32.794597 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 27105
I20260925 10:10:32.800797 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 26718
I20260925 10:10:32.806226 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:37941
--webserver_interface=127.14.41.126
--webserver_port=33125
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:37941
--ranger_config_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.41.84:32929/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.41.84:32929/kms with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:10:32.890220 27237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:32.890441 27237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:32.890463 27237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:32.891883 27237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:10:32.891978 27237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:32.892005 27237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:32.892027 27237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:10:32.892046 27237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:10:32.894780 27237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.41.84:32929/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:37941
--ranger_config_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:37941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=33125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:32.895094 27237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:32.895377 27237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:32.901782 27245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:32.901762 27243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:32.901794 27242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:32.902163 27237 server_base.cc:1034] running on GCE node
I20260925 10:10:32.902405 27237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:32.902697 27237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:32.903858 27237 hybrid_clock.cc:648] HybridClock initialized: now 1790331032903849 us; error 37 us; skew 500 ppm
Sep 25 10:10:32 dist-test-slave-s6tm krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331032, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:32.906375 27237 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:10:32.906814 27237 webserver.cc:460] Webserver started at http://127.14.41.126:33125/ using document root <none> and password file <none>
I20260925 10:10:32.907034 27237 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:32.907109 27237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:10:32 dist-test-slave-s6tm krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331032, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for HTTP/127.14.41.84@KRBTEST.COM
I20260925 10:10:32.948863 27237 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:10:32.949615 27253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:32.949797 27237 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:10:32.949896 27237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "f9b002b10bcd4e62bc7c51a7f2b2fde0"
format_stamp: "Formatted at 2026-09-25 10:10:31 on dist-test-slave-s6tm"
server_key: "35dd4e3312bb55f57f7d2e526579e28e"
server_key_iv: "72cb3f52fa73f0a23966f939f189a771"
server_key_version: "kuduclusterkey@0"
I20260925 10:10:32.950290 27237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:32.960062 27237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:32.961894 27237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:32.962036 27237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:32.966238 27237 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:37941
I20260925 10:10:32.966275 27305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:37941 every 8 connection(s)
I20260925 10:10:32.966674 27237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:10:32.967984 27306 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:10:32.968659 27306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0: Bootstrap starting.
I20260925 10:10:32.972190 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 27237
I20260925 10:10:32.972482 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.41.84:32929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:42813
--local_ip_for_outbound_sockets=127.14.41.65
--tserver_master_addrs=127.14.41.126:37941
--webserver_port=44705
--webserver_interface=127.14.41.65
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.41.84:32929/kms with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:10:32.977083 27306 log.cc:824] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0: Log is configured to *not* fsync() on all Append() calls
I20260925 10:10:32.978952 27306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:10:32.979278 27306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0: Bootstrap complete.
I20260925 10:10:32.981791 27306 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b002b10bcd4e62bc7c51a7f2b2fde0" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37941 } }
I20260925 10:10:32.982080 27306 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b002b10bcd4e62bc7c51a7f2b2fde0, State: Initialized, Role: FOLLOWER
I20260925 10:10:32.982280 27306 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b002b10bcd4e62bc7c51a7f2b2fde0" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37941 } }
I20260925 10:10:32.982366 27306 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:10:32.982421 27306 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:10:32.982470 27306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:10:32.983527 27306 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b002b10bcd4e62bc7c51a7f2b2fde0" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37941 } }
I20260925 10:10:32.983654 27306 leader_election.cc:302] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9b002b10bcd4e62bc7c51a7f2b2fde0; no voters:
I20260925 10:10:32.983868 27306 leader_election.cc:288] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:10:32.983901 27310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:10:32.984040 27310 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [term 2 LEADER]: Becoming Leader. State: Replica: f9b002b10bcd4e62bc7c51a7f2b2fde0, State: Running, Role: LEADER
I20260925 10:10:32.984148 27310 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b002b10bcd4e62bc7c51a7f2b2fde0" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37941 } }
I20260925 10:10:32.984176 27306 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:10:32.984642 27311 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9b002b10bcd4e62bc7c51a7f2b2fde0. Latest consensus state: current_term: 2 leader_uuid: "f9b002b10bcd4e62bc7c51a7f2b2fde0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b002b10bcd4e62bc7c51a7f2b2fde0" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37941 } } }
I20260925 10:10:32.984721 27311 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [sys.catalog]: This master's current role is: LEADER
I20260925 10:10:32.984995 27315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:10:32.985476 27315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:10:32.985586 27315 catalog_manager.cc:1259] Loaded cluster ID: a0391849042746a285bfd1bb68556d69
I20260925 10:10:32.985613 27315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:10:32.986157 27306 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 10:10:32.986733 27315 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:10:32.986927 27315 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0: Loaded TSK: 0
I20260925 10:10:32.987147 27315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:10:32.987108 27312 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f9b002b10bcd4e62bc7c51a7f2b2fde0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b002b10bcd4e62bc7c51a7f2b2fde0" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 37941 } } }
I20260925 10:10:32.987339 27312 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0 [sys.catalog]: This master's current role is: LEADER
W20260925 10:10:33.093852 27308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:33.094177 27308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:33.094223 27308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:33.095800 27308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:33.095871 27308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:33.095916 27308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:10:33.098158 27308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.41.84:32929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:42813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=44705
--tserver_master_addrs=127.14.41.126:37941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:33.098448 27308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:33.098702 27308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:33.105079 27343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:33.105085 27346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:33.105557 27344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:33.106021 27308 server_base.cc:1034] running on GCE node
I20260925 10:10:33.106357 27308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:33.106679 27308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:33.114097 27308 hybrid_clock.cc:648] HybridClock initialized: now 1790331033114055 us; error 54 us; skew 500 ppm
Sep 25 10:10:33 dist-test-slave-s6tm krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331033, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:33.118453 27308 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:10:33.119046 27308 webserver.cc:460] Webserver started at http://127.14.41.65:44705/ using document root <none> and password file <none>
I20260925 10:10:33.119318 27308 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:33.119457 27308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:10:33 dist-test-slave-s6tm krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331033, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for HTTP/127.14.41.84@KRBTEST.COM
I20260925 10:10:33.149071 27308 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:10:33.150105 27354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:33.150285 27308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:10:33.150384 27308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "a91da2248cd249338c53668a08276a6c"
format_stamp: "Formatted at 2026-09-25 10:10:32 on dist-test-slave-s6tm"
server_key: "d82d9a8add27ad9c721585a6e580aa8c"
server_key_iv: "febde9dea3c43d0b237dd7db85acee7b"
server_key_version: "kuduclusterkey@0"
I20260925 10:10:33.150907 27308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:33.174865 27308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:33.177435 27308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:33.177606 27308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:33.177884 27308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:10:33.178300 27308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:10:33.178356 27308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:33.178390 27308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:10:33.178416 27308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:33.185303 27308 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:42813
I20260925 10:10:33.185812 27308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:10:33.186936 27467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:42813 every 8 connection(s)
I20260925 10:10:33.194564 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 27308
I20260925 10:10:33.194927 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.41.84:32929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:41239
--local_ip_for_outbound_sockets=127.14.41.66
--tserver_master_addrs=127.14.41.126:37941
--webserver_port=40849
--webserver_interface=127.14.41.66
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.41.84:32929/kms with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:10:33 dist-test-slave-s6tm krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331033, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:10:33.202773 27470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:33.187845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:37941 (local address 127.14.41.65:34087)
0925 10:10:33.188782 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:33.188823 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:33.189486 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:33.190305 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:33.190309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:33.190592 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:33.190868 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:33.190875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:33.192978 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:33.192980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:33.193865 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:33.193868 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:33.194837 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:33.194839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:33.195009 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:33.195694 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:33.195706 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:33.197593 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:33.200205 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:33.200215 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:33.200218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:33.200638 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:33.201054 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:33.201055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:33.201056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:33.201235 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:33.201879 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:33.201887 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:33.202101 (+ 214us) client_negotiation.cc:769] Sending connection context
0925 10:10:33.202217 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 10:10:33.202432 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"thread_start_us":131,"threads_started":1}
I20260925 10:10:33.203827 27471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:33.187994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:34087 (local address 127.14.41.126:37941)
0925 10:10:33.189729 (+ 1735us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:33.189739 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:33.189780 (+ 41us) server_negotiation.cc:407] Connection header received
0925 10:10:33.189936 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:33.189942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:33.190001 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:33.190136 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:33.191036 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:33.192761 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:33.193987 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:33.194572 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:33.194694 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:33.197777 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:33.197784 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:33.197784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:33.197799 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:33.200032 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:33.200773 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:33.200779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:33.200779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:33.200928 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:33.201350 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:33.201351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:33.201351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:33.201769 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:33.203302 (+ 1533us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:33.203396 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 10:10:33.203561 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1667,"thread_start_us":88,"threads_started":1}
I20260925 10:10:33.204630 27468 heartbeater.cc:347] Connected to a master server at 127.14.41.126:37941
I20260925 10:10:33.204746 27468 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:33.205026 27468 heartbeater.cc:511] Master 127.14.41.126:37941 requested a full tablet report, sending...
W20260925 10:10:33.205927 27468 heartbeater.cc:644] Failed to heartbeat to 127.14.41.126:37941 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 10:10:33.206010 27468 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:33.206091 27468 heartbeater.cc:511] Master 127.14.41.126:37941 requested a full tablet report, sending...
I20260925 10:10:33.206432 27468 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:33.206522 27468 heartbeater.cc:511] Master 127.14.41.126:37941 requested a full tablet report, sending...
W20260925 10:10:33.206835 27468 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:10:33.302971 27472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:33.303254 27472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:33.303292 27472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:33.305328 27472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:33.305413 27472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:33.305459 27472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:10:33.307793 27472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.41.84:32929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:41239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=40849
--tserver_master_addrs=127.14.41.126:37941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:33.308103 27472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:33.308395 27472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:33.313928 27477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:33.314152 27472 server_base.cc:1034] running on GCE node
W20260925 10:10:33.314316 27480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:33.314460 27478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:33.314697 27472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:33.314985 27472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:33.316144 27472 hybrid_clock.cc:648] HybridClock initialized: now 1790331033316133 us; error 27 us; skew 500 ppm
Sep 25 10:10:33 dist-test-slave-s6tm krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331033, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:33.318833 27472 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:10:33.319334 27472 webserver.cc:460] Webserver started at http://127.14.41.66:40849/ using document root <none> and password file <none>
I20260925 10:10:33.319615 27472 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:33.319697 27472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:10:33 dist-test-slave-s6tm krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331033, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for HTTP/127.14.41.84@KRBTEST.COM
I20260925 10:10:33.339342 27472 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:10:33.342825 27488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:33.343127 27472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:10:33.343230 27472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "89faffa2bb504b568f49fb1e19fc34ad"
format_stamp: "Formatted at 2026-09-25 10:10:32 on dist-test-slave-s6tm"
server_key: "efa0bed1e6966236559f1a50c6619284"
server_key_iv: "e33063587963000d3a50c0952d7a7a46"
server_key_version: "kuduclusterkey@0"
I20260925 10:10:33.343719 27472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:33.360883 27472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:33.363170 27472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:33.363338 27472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:33.363590 27472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:10:33.363922 27472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:10:33.364033 27472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:33.364074 27472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:10:33.364100 27472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:33.371420 27472 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:41239
I20260925 10:10:33.371786 27601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:41239 every 8 connection(s)
I20260925 10:10:33.371964 27472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:10:33.372963 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 27472
I20260925 10:10:33.373289 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.14.41.84:32929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:42351
--local_ip_for_outbound_sockets=127.14.41.67
--tserver_master_addrs=127.14.41.126:37941
--webserver_port=41075
--webserver_interface=127.14.41.67
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.14.41.84:32929/kms with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:10:33 dist-test-slave-s6tm krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331033, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:10:33.387794 27471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:33.374072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:37935 (local address 127.14.41.126:37941)
0925 10:10:33.374168 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:33.374170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:33.375165 (+ 995us) server_negotiation.cc:407] Connection header received
0925 10:10:33.376600 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:33.376602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:33.376622 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:33.376713 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:33.377516 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:33.378409 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:33.380203 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:33.380557 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:33.380644 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:33.383546 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:33.383548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:33.383549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:33.383559 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:33.385575 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:33.386111 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:33.386113 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:33.386114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:33.386188 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:33.386503 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:33.386504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:33.386504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:33.386677 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:33.386771 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:33.387557 (+ 786us) server_negotiation.cc:299] Negotiation successful
0925 10:10:33.387676 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:10:33.388455 27604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:33.373773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:37941 (local address 127.14.41.66:37935)
0925 10:10:33.374968 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:33.375029 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:33.376446 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:33.376816 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:33.376822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:33.377084 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:33.377382 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:33.377390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:33.378576 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:33.378577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:33.380090 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:33.380093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:33.380750 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:33.380752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:33.380933 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:33.381681 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:33.381693 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:33.383414 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:33.385667 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:33.385676 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:33.385679 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:33.386012 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:33.386276 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:33.386277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:33.386277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:33.386424 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:33.386765 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:33.386770 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:33.386943 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 10:10:33.387506 (+ 563us) client_negotiation.cc:240] Negotiation successful
0925 10:10:33.388206 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1145,"thread_start_us":78,"threads_started":1}
I20260925 10:10:33.389356 27602 heartbeater.cc:347] Connected to a master server at 127.14.41.126:37941
I20260925 10:10:33.389468 27602 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:33.389681 27602 heartbeater.cc:511] Master 127.14.41.126:37941 requested a full tablet report, sending...
W20260925 10:10:33.390375 27602 heartbeater.cc:644] Failed to heartbeat to 127.14.41.126:37941 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 10:10:33.390452 27602 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:33.390511 27602 heartbeater.cc:511] Master 127.14.41.126:37941 requested a full tablet report, sending...
I20260925 10:10:33.390910 27602 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:33.391000 27602 heartbeater.cc:511] Master 127.14.41.126:37941 requested a full tablet report, sending...
W20260925 10:10:33.391359 27602 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:10:33.481647 27605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:33.481966 27605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:33.482013 27605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:33.483995 27605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:33.484086 27605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:33.484131 27605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:10:33.486531 27605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.14.41.84:32929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:42351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=41075
--tserver_master_addrs=127.14.41.126:37941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:33.486812 27605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:33.487114 27605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:33.495079 27610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:33.495157 27611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:33.495450 27613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:33.496155 27605 server_base.cc:1034] running on GCE node
I20260925 10:10:33.496426 27605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:33.496760 27605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:33.497951 27605 hybrid_clock.cc:648] HybridClock initialized: now 1790331033497936 us; error 29 us; skew 500 ppm
Sep 25 10:10:33 dist-test-slave-s6tm krb5kdc[26274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331033, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:33.501483 27605 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:10:33.502097 27605 webserver.cc:460] Webserver started at http://127.14.41.67:41075/ using document root <none> and password file <none>
I20260925 10:10:33.502413 27605 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:33.502502 27605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:10:33 dist-test-slave-s6tm krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331033, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for HTTP/127.14.41.84@KRBTEST.COM
I20260925 10:10:33.524536 27605 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260925 10:10:33.525863 27621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:33.526055 27605 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:10:33.526160 27605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "b67c9524c1d749cdbce11cd22752eb7e"
format_stamp: "Formatted at 2026-09-25 10:10:32 on dist-test-slave-s6tm"
server_key: "f20e40ed1362b2da1f02f95335ebc85e"
server_key_iv: "c038796cc9c19d6568f56de76d983e87"
server_key_version: "kuduclusterkey@0"
I20260925 10:10:33.526732 27605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:33.542194 27605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:33.545080 27605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:33.545264 27605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:33.545552 27605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:10:33.545958 27605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:10:33.546020 27605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:33.546064 27605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:10:33.546090 27605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:33.553750 27605 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:42351
I20260925 10:10:33.554291 27605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:10:33.554556 27735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:42351 every 8 connection(s)
I20260925 10:10:33.560503 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 27605
Sep 25 10:10:33 dist-test-slave-s6tm krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331033, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:10:33.565766 27471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:33.555403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:45847 (local address 127.14.41.126:37941)
0925 10:10:33.555460 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:33.555463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:33.555767 (+ 304us) server_negotiation.cc:407] Connection header received
0925 10:10:33.556458 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:33.556460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:33.556473 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:33.556566 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:33.557147 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:33.558001 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:33.559080 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:33.559435 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:33.559520 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:33.562082 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:33.562084 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:33.562085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:33.562093 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:33.563900 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:33.564525 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:33.564527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:33.564527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:33.564600 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:33.564918 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:33.564919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:33.564919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:33.565081 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:33.565157 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:33.565563 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 10:10:33.565660 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:10:33.566380 27738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:33.555344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:37941 (local address 127.14.41.67:45847)
0925 10:10:33.555631 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:33.555673 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:33.556331 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:33.556658 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:33.556662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:33.556843 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:33.557044 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:33.557048 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:33.558107 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:33.558107 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:33.558980 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:33.558982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:33.559578 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:33.559579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:33.559681 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:33.560352 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:33.560364 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:33.561966 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:33.564024 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:33.564033 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:33.564035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:33.564407 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:33.564686 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:33.564687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:33.564687 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:33.564846 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:33.565162 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:33.565169 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:33.565376 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 10:10:33.565910 (+ 534us) client_negotiation.cc:240] Negotiation successful
0925 10:10:33.566102 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":86,"threads_started":1}
I20260925 10:10:33.566884 27736 heartbeater.cc:347] Connected to a master server at 127.14.41.126:37941
I20260925 10:10:33.567006 27736 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:33.567224 27736 heartbeater.cc:511] Master 127.14.41.126:37941 requested a full tablet report, sending...
W20260925 10:10:33.568040 27736 heartbeater.cc:644] Failed to heartbeat to 127.14.41.126:37941 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 10:10:33.568120 27736 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:33.568187 27736 heartbeater.cc:511] Master 127.14.41.126:37941 requested a full tablet report, sending...
I20260925 10:10:33.568665 27736 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:33.568744 27736 heartbeater.cc:511] Master 127.14.41.126:37941 requested a full tablet report, sending...
I20260925 10:10:33.569552 27739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:33.560982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48566 (local address 127.14.41.126:37941)
0925 10:10:33.561114 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:33.561117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:33.561246 (+ 129us) server_negotiation.cc:407] Connection header received
0925 10:10:33.561325 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:33.561326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:33.561339 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:33.561421 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:33.562278 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:33.562975 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:33.564189 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:33.564406 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:33.564476 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:33.565906 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:33.565907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:33.565907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:33.565914 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:33.567982 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:33.568388 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:33.568389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:33.568390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:33.568444 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:33.568637 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:33.568638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:33.568638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:33.568770 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:33.568933 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:33.569349 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 10:10:33.569435 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
W20260925 10:10:33.572078 27736 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 10:10:34 dist-test-slave-s6tm krb5kdc[26274](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790331034, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:34.207072 27468 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:34.207255 27468 heartbeater.cc:511] Master 127.14.41.126:37941 requested a full tablet report, sending...
I20260925 10:10:34.207619 27270 ts_manager.cc:194] Registered new tserver with Master: a91da2248cd249338c53668a08276a6c (127.14.41.65:42813)
I20260925 10:10:34.208390 27270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:34087
I20260925 10:10:34.391575 27602 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:34.391729 27602 heartbeater.cc:511] Master 127.14.41.126:37941 requested a full tablet report, sending...
I20260925 10:10:34.392155 27270 ts_manager.cc:194] Registered new tserver with Master: 89faffa2bb504b568f49fb1e19fc34ad (127.14.41.66:41239)
I20260925 10:10:34.392833 27270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:37935
Sep 25 10:10:34 dist-test-slave-s6tm krb5kdc[26274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331034, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for HTTP/127.14.41.84@KRBTEST.COM
I20260925 10:10:34.572263 27736 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:34.572449 27736 heartbeater.cc:511] Master 127.14.41.126:37941 requested a full tablet report, sending...
I20260925 10:10:34.572784 27270 ts_manager.cc:194] Registered new tserver with Master: b67c9524c1d749cdbce11cd22752eb7e (127.14.41.67:42351)
I20260925 10:10:34.573195 27270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:45847
I20260925 10:10:34.579334 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:10:34.606922 27757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:34.581112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.14.41.126:37941)
0925 10:10:34.581419 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:34.581421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:34.584157 (+ 2736us) server_negotiation.cc:407] Connection header received
0925 10:10:34.584190 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:34.584191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:34.584204 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:34.584261 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:34.585142 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.586031 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.587082 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.587394 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.587450 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:34.591287 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:34.591289 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:34.591289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:34.591298 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:34.605034 (+ 13736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:34.605572 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:34.605574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:34.605574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:34.605649 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:34.605897 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:34.605898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:34.605898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:34.606064 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:34.606168 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:34.606703 (+ 535us) server_negotiation.cc:299] Negotiation successful
0925 10:10:34.606803 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":73,"threads_started":1}
I20260925 10:10:34.609310 27270 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:10:34.611764 27270 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:10:34.637559 27766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:34.621696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:42351 (local address 127.0.0.1:49402)
0925 10:10:34.623279 (+ 1583us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:34.623327 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:34.623396 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:34.623974 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:34.623977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:34.623980 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:10:34.624060 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:34.624062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.633308 (+ 9246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.633309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:34.634857 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:34.634861 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.637137 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.637138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:34.637253 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:34.637273 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 10:10:34.637360 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 10:10:34.637425 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1547,"thread_start_us":73,"threads_started":1}
I20260925 10:10:34.640684 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:34.622094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:41239 (local address 127.0.0.1:40558)
0925 10:10:34.629247 (+ 7153us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:34.629258 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:34.629321 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:34.629653 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:34.629654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:34.629664 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:10:34.629715 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:34.629717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.631700 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.631701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:34.638416 (+ 6715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:34.638420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.640326 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.640328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:34.640458 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:34.640464 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:10:34.640529 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 10:10:34.640583 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7134,"thread_start_us":70,"threads_started":1}
I20260925 10:10:34.641782 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:34.622492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.14.41.67:42351)
0925 10:10:34.623534 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:34.623540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:34.623550 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:10:34.623589 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:34.623591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:34.623608 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:34.623685 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:10:34.624155 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.633194 (+ 9039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.635995 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.636918 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.637019 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:34.637093 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:34.641623 (+ 4530us) server_negotiation.cc:299] Negotiation successful
0925 10:10:34.641687 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1008,"thread_start_us":53,"threads_started":1}
I20260925 10:10:34.642233 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:34.622791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40558 (local address 127.14.41.66:41239)
0925 10:10:34.623907 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:34.623911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:34.629386 (+ 5475us) server_negotiation.cc:407] Connection header received
0925 10:10:34.629432 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:34.629435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:34.629452 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:34.629549 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:10:34.629802 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.631358 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.638535 (+ 7177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.639491 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.639573 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:34.639658 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:34.642085 (+ 2427us) server_negotiation.cc:299] Negotiation successful
0925 10:10:34.642147 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1073,"thread_start_us":55,"threads_started":1}
I20260925 10:10:34.642613 27768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:34.622637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58274 (local address 127.14.41.65:42813)
0925 10:10:34.623824 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:34.623829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:34.625326 (+ 1497us) server_negotiation.cc:407] Connection header received
0925 10:10:34.625368 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:34.625371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:34.625388 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:34.625472 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:10:34.625814 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.627052 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.635021 (+ 7969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.641070 (+ 6049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.641150 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:34.641226 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:34.642465 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0925 10:10:34.642529 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1156,"thread_start_us":66,"threads_started":1}
I20260925 10:10:34.641531 27765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:34.621950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:42813 (local address 127.0.0.1:58274)
0925 10:10:34.625186 (+ 3236us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:34.625197 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:34.625255 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:34.625661 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:34.625662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:34.625665 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:10:34.625731 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:34.625733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.627511 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.627512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:34.629126 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:34.629130 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.641266 (+ 12136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.641267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:34.641355 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:34.641359 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:10:34.641412 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 10:10:34.641459 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3218,"thread_start_us":68,"threads_started":1}
I20260925 10:10:34.644373 27670 tablet_service.cc:1467] Processing CreateTablet for tablet b8c14195a0f5442caa8104734d0348b4 (DEFAULT_TABLE table=test-table [id=0788976e47f94c3aac3bbb3bdc119a5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:10:34.644562 27520 tablet_service.cc:1467] Processing CreateTablet for tablet b8c14195a0f5442caa8104734d0348b4 (DEFAULT_TABLE table=test-table [id=0788976e47f94c3aac3bbb3bdc119a5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:10:34.644779 27670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c14195a0f5442caa8104734d0348b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:10:34.644968 27520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c14195a0f5442caa8104734d0348b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:10:34.652786 27402 tablet_service.cc:1467] Processing CreateTablet for tablet b8c14195a0f5442caa8104734d0348b4 (DEFAULT_TABLE table=test-table [id=0788976e47f94c3aac3bbb3bdc119a5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:10:34.653192 27402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c14195a0f5442caa8104734d0348b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:10:34.657553 27770 tablet_bootstrap.cc:492] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e: Bootstrap starting.
I20260925 10:10:34.658349 27770 tablet_bootstrap.cc:654] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e: Neither blocks nor log segments found. Creating new log.
I20260925 10:10:34.658747 27770 log.cc:824] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:10:34.660136 27772 tablet_bootstrap.cc:492] T b8c14195a0f5442caa8104734d0348b4 P a91da2248cd249338c53668a08276a6c: Bootstrap starting.
I20260925 10:10:34.660878 27772 tablet_bootstrap.cc:654] T b8c14195a0f5442caa8104734d0348b4 P a91da2248cd249338c53668a08276a6c: Neither blocks nor log segments found. Creating new log.
I20260925 10:10:34.661275 27772 log.cc:824] T b8c14195a0f5442caa8104734d0348b4 P a91da2248cd249338c53668a08276a6c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:10:34.661677 27771 tablet_bootstrap.cc:492] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad: Bootstrap starting.
I20260925 10:10:34.662410 27771 tablet_bootstrap.cc:654] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad: Neither blocks nor log segments found. Creating new log.
I20260925 10:10:34.662778 27771 log.cc:824] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad: Log is configured to *not* fsync() on all Append() calls
I20260925 10:10:34.664734 27772 tablet_bootstrap.cc:492] T b8c14195a0f5442caa8104734d0348b4 P a91da2248cd249338c53668a08276a6c: No bootstrap required, opened a new log
I20260925 10:10:34.664860 27772 ts_tablet_manager.cc:1397] T b8c14195a0f5442caa8104734d0348b4 P a91da2248cd249338c53668a08276a6c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 10:10:34.668412 27772 raft_consensus.cc:348] T b8c14195a0f5442caa8104734d0348b4 P a91da2248cd249338c53668a08276a6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67c9524c1d749cdbce11cd22752eb7e" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42351 } } peers { permanent_uuid: "a91da2248cd249338c53668a08276a6c" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42813 } } peers { permanent_uuid: "89faffa2bb504b568f49fb1e19fc34ad" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 41239 } }
I20260925 10:10:34.668866 27772 raft_consensus.cc:374] T b8c14195a0f5442caa8104734d0348b4 P a91da2248cd249338c53668a08276a6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:10:34.669046 27772 raft_consensus.cc:729] T b8c14195a0f5442caa8104734d0348b4 P a91da2248cd249338c53668a08276a6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91da2248cd249338c53668a08276a6c, State: Initialized, Role: FOLLOWER
I20260925 10:10:34.669498 27772 consensus_queue.cc:260] T b8c14195a0f5442caa8104734d0348b4 P a91da2248cd249338c53668a08276a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67c9524c1d749cdbce11cd22752eb7e" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42351 } } peers { permanent_uuid: "a91da2248cd249338c53668a08276a6c" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42813 } } peers { permanent_uuid: "89faffa2bb504b568f49fb1e19fc34ad" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 41239 } }
I20260925 10:10:34.669942 27772 ts_tablet_manager.cc:1428] T b8c14195a0f5442caa8104734d0348b4 P a91da2248cd249338c53668a08276a6c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 10:10:34.670156 27468 heartbeater.cc:503] Master 127.14.41.126:37941 was elected leader, sending a full tablet report...
I20260925 10:10:34.672025 27770 tablet_bootstrap.cc:492] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e: No bootstrap required, opened a new log
I20260925 10:10:34.672108 27770 ts_tablet_manager.cc:1397] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260925 10:10:34.674703 27770 raft_consensus.cc:348] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67c9524c1d749cdbce11cd22752eb7e" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42351 } } peers { permanent_uuid: "a91da2248cd249338c53668a08276a6c" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42813 } } peers { permanent_uuid: "89faffa2bb504b568f49fb1e19fc34ad" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 41239 } }
I20260925 10:10:34.674893 27770 raft_consensus.cc:374] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:10:34.674924 27770 raft_consensus.cc:729] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67c9524c1d749cdbce11cd22752eb7e, State: Initialized, Role: FOLLOWER
I20260925 10:10:34.675225 27770 consensus_queue.cc:260] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67c9524c1d749cdbce11cd22752eb7e" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42351 } } peers { permanent_uuid: "a91da2248cd249338c53668a08276a6c" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42813 } } peers { permanent_uuid: "89faffa2bb504b568f49fb1e19fc34ad" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 41239 } }
I20260925 10:10:34.675588 27770 ts_tablet_manager.cc:1428] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:10:34.675753 27736 heartbeater.cc:503] Master 127.14.41.126:37941 was elected leader, sending a full tablet report...
I20260925 10:10:34.677274 27771 tablet_bootstrap.cc:492] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad: No bootstrap required, opened a new log
I20260925 10:10:34.677351 27771 ts_tablet_manager.cc:1397] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20260925 10:10:34.680812 27771 raft_consensus.cc:348] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67c9524c1d749cdbce11cd22752eb7e" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42351 } } peers { permanent_uuid: "a91da2248cd249338c53668a08276a6c" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42813 } } peers { permanent_uuid: "89faffa2bb504b568f49fb1e19fc34ad" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 41239 } }
I20260925 10:10:34.681066 27771 raft_consensus.cc:374] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:10:34.681143 27771 raft_consensus.cc:729] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89faffa2bb504b568f49fb1e19fc34ad, State: Initialized, Role: FOLLOWER
I20260925 10:10:34.681334 27771 consensus_queue.cc:260] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67c9524c1d749cdbce11cd22752eb7e" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42351 } } peers { permanent_uuid: "a91da2248cd249338c53668a08276a6c" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42813 } } peers { permanent_uuid: "89faffa2bb504b568f49fb1e19fc34ad" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 41239 } }
I20260925 10:10:34.681710 27771 ts_tablet_manager.cc:1428] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 10:10:34.681862 27602 heartbeater.cc:503] Master 127.14.41.126:37941 was elected leader, sending a full tablet report...
W20260925 10:10:34.688684 27469 tablet.cc:2369] T b8c14195a0f5442caa8104734d0348b4 P a91da2248cd249338c53668a08276a6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:10:34.689198 27778 raft_consensus.cc:482] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:10:34.689392 27778 raft_consensus.cc:504] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67c9524c1d749cdbce11cd22752eb7e" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42351 } } peers { permanent_uuid: "a91da2248cd249338c53668a08276a6c" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42813 } } peers { permanent_uuid: "89faffa2bb504b568f49fb1e19fc34ad" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 41239 } }
I20260925 10:10:34.690732 27778 leader_election.cc:288] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b67c9524c1d749cdbce11cd22752eb7e (127.14.41.67:42351), a91da2248cd249338c53668a08276a6c (127.14.41.65:42813)
I20260925 10:10:34.695094 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:34.690284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:40855 (local address 127.14.41.67:42351)
0925 10:10:34.690330 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:34.690333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:34.690988 (+ 655us) server_negotiation.cc:407] Connection header received
0925 10:10:34.691092 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:34.691093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:34.691102 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:34.691131 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:10:34.691349 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.692149 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.693641 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.694204 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.694252 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:34.694260 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:34.694962 (+ 702us) server_negotiation.cc:299] Negotiation successful
0925 10:10:34.695018 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:10:34.695564 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:34.690052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:42351 (local address 127.14.41.66:40855)
0925 10:10:34.690919 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:34.690931 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:34.691017 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:34.691197 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:34.691198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:34.691199 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:10:34.691281 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:34.691283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.692246 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.692246 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:34.693530 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:34.693533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.694352 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.694353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:34.694434 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:34.694438 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:10:34.695417 (+ 979us) client_negotiation.cc:240] Negotiation successful
0925 10:10:34.695467 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":100,"threads_started":1}
I20260925 10:10:34.696137 27690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c14195a0f5442caa8104734d0348b4" candidate_uuid: "89faffa2bb504b568f49fb1e19fc34ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67c9524c1d749cdbce11cd22752eb7e" is_pre_election: true
I20260925 10:10:34.696300 27690 raft_consensus.cc:2454] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89faffa2bb504b568f49fb1e19fc34ad in term 0.
I20260925 10:10:34.696565 27489 leader_election.cc:302] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89faffa2bb504b568f49fb1e19fc34ad, b67c9524c1d749cdbce11cd22752eb7e; no voters:
I20260925 10:10:34.696798 27778 raft_consensus.cc:2781] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:10:34.696892 27778 raft_consensus.cc:482] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:10:34.696956 27778 raft_consensus.cc:3037] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:10:34.698096 27778 raft_consensus.cc:504] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67c9524c1d749cdbce11cd22752eb7e" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42351 } } peers { permanent_uuid: "a91da2248cd249338c53668a08276a6c" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42813 } } peers { permanent_uuid: "89faffa2bb504b568f49fb1e19fc34ad" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 41239 } }
I20260925 10:10:34.698452 27690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c14195a0f5442caa8104734d0348b4" candidate_uuid: "89faffa2bb504b568f49fb1e19fc34ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67c9524c1d749cdbce11cd22752eb7e"
I20260925 10:10:34.698534 27690 raft_consensus.cc:3037] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:10:34.699347 27690 raft_consensus.cc:2454] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89faffa2bb504b568f49fb1e19fc34ad in term 1.
I20260925 10:10:34.699580 27489 leader_election.cc:302] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89faffa2bb504b568f49fb1e19fc34ad, b67c9524c1d749cdbce11cd22752eb7e; no voters:
I20260925 10:10:34.699838 27778 leader_election.cc:288] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [CANDIDATE]: Term 1 election: Requested vote from peers b67c9524c1d749cdbce11cd22752eb7e (127.14.41.67:42351), a91da2248cd249338c53668a08276a6c (127.14.41.65:42813)
I20260925 10:10:34.699919 27778 raft_consensus.cc:2781] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:10:34.700016 27778 raft_consensus.cc:686] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [term 1 LEADER]: Becoming Leader. State: Replica: 89faffa2bb504b568f49fb1e19fc34ad, State: Running, Role: LEADER
I20260925 10:10:34.700212 27778 consensus_queue.cc:237] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67c9524c1d749cdbce11cd22752eb7e" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42351 } } peers { permanent_uuid: "a91da2248cd249338c53668a08276a6c" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42813 } } peers { permanent_uuid: "89faffa2bb504b568f49fb1e19fc34ad" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 41239 } }
I20260925 10:10:34.703683 27270 catalog_manager.cc:5495] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 89faffa2bb504b568f49fb1e19fc34ad (127.14.41.66). New cstate: current_term: 1 leader_uuid: "89faffa2bb504b568f49fb1e19fc34ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67c9524c1d749cdbce11cd22752eb7e" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a91da2248cd249338c53668a08276a6c" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89faffa2bb504b568f49fb1e19fc34ad" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 41239 } health_report { overall_health: HEALTHY } } }
I20260925 10:10:34.705950 27768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:34.690674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:46019 (local address 127.14.41.65:42813)
0925 10:10:34.690713 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:34.690715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:34.700151 (+ 9436us) server_negotiation.cc:407] Connection header received
0925 10:10:34.700239 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:34.700241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:34.700252 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:34.700310 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:10:34.701130 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.701906 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.704906 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.705535 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.705586 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:34.705595 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:34.705826 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 10:10:34.705873 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:10:34.706370 27780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:34.690496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:42813 (local address 127.14.41.66:46019)
0925 10:10:34.700049 (+ 9553us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:34.700064 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:34.700164 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:34.700641 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:34.700642 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:34.700644 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:10:34.700706 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:34.700708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.702007 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.702008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:34.703153 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:34.703156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.705655 (+ 2499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.705656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:34.705746 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:34.705750 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:10:34.706216 (+ 466us) client_negotiation.cc:240] Negotiation successful
0925 10:10:34.706265 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9525,"thread_start_us":74,"threads_started":1}
I20260925 10:10:34.706883 27416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c14195a0f5442caa8104734d0348b4" candidate_uuid: "89faffa2bb504b568f49fb1e19fc34ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91da2248cd249338c53668a08276a6c" is_pre_election: true
I20260925 10:10:34.707036 27416 raft_consensus.cc:2454] T b8c14195a0f5442caa8104734d0348b4 P a91da2248cd249338c53668a08276a6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89faffa2bb504b568f49fb1e19fc34ad in term 0.
I20260925 10:10:34.707134 27417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c14195a0f5442caa8104734d0348b4" candidate_uuid: "89faffa2bb504b568f49fb1e19fc34ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91da2248cd249338c53668a08276a6c"
I20260925 10:10:34.707175 27417 raft_consensus.cc:3037] T b8c14195a0f5442caa8104734d0348b4 P a91da2248cd249338c53668a08276a6c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:10:34.708387 27417 raft_consensus.cc:2454] T b8c14195a0f5442caa8104734d0348b4 P a91da2248cd249338c53668a08276a6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89faffa2bb504b568f49fb1e19fc34ad in term 1.
I20260925 10:10:34.718676 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:34.715355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40572 (local address 127.14.41.66:41239)
0925 10:10:34.715404 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:34.715406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:34.715413 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:10:34.715448 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:34.715449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:34.715468 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:34.715528 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:10:34.715710 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.716488 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.717594 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:34.717904 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:34.717962 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:34.718153 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:10:34.718284 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:10:34.718353 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:34.718575 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 10:10:34.718596 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 10:10:34.721082 27417 raft_consensus.cc:1261] T b8c14195a0f5442caa8104734d0348b4 P a91da2248cd249338c53668a08276a6c [term 1 FOLLOWER]: Refusing update from remote peer 89faffa2bb504b568f49fb1e19fc34ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:10:34.721526 27781 consensus_queue.cc:1035] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91da2248cd249338c53668a08276a6c" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:10:34.723767 27690 raft_consensus.cc:1261] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e [term 1 FOLLOWER]: Refusing update from remote peer 89faffa2bb504b568f49fb1e19fc34ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:10:34.724164 27781 consensus_queue.cc:1035] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67c9524c1d749cdbce11cd22752eb7e" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:10:34.728295 27784 mvcc.cc:205] Tried to move back new op lower bound from 7333195918214266880 to 7333195918134419456. Current Snapshot: MvccSnapshot[applied={T|T < 7333195918214266880}]
I20260925 10:10:34.729123 27787 mvcc.cc:205] Tried to move back new op lower bound from 7333195918214266880 to 7333195918134419456. Current Snapshot: MvccSnapshot[applied={T|T < 7333195918214266880}]
I20260925 10:10:34.734074 27270 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:10:34.734184 27270 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:10:34.735101 27270 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9b002b10bcd4e62bc7c51a7f2b2fde0: Sending DeleteTablet for 3 replicas of tablet b8c14195a0f5442caa8104734d0348b4
I20260925 10:10:34.735522 27402 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c14195a0f5442caa8104734d0348b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:10:34 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:58274
I20260925 10:10:34.735891 27670 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c14195a0f5442caa8104734d0348b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:10:34 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:49402
I20260925 10:10:34.736320 27520 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c14195a0f5442caa8104734d0348b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:10:34 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:40558
I20260925 10:10:34.736934 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 27308
I20260925 10:10:34.744077 27793 tablet_replica.cc:330] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e: stopping tablet replica
I20260925 10:10:34.744273 27794 tablet_replica.cc:330] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad: stopping tablet replica
I20260925 10:10:34.744478 27794 raft_consensus.cc:2229] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:10:34.744632 27793 raft_consensus.cc:2229] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:10:34.744834 27794 raft_consensus.cc:2258] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:10:34.745003 27793 raft_consensus.cc:2258] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:10:34.745656 27794 ts_tablet_manager.cc:1905] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:10:34.746076 27793 ts_tablet_manager.cc:1905] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:10:34.751375 27793 ts_tablet_manager.cc:1918] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:10:34.751605 27794 ts_tablet_manager.cc:1918] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:10:34.751724 27794 log.cc:1196] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/b8c14195a0f5442caa8104734d0348b4
I20260925 10:10:34.751997 27793 log.cc:1196] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/b8c14195a0f5442caa8104734d0348b4
I20260925 10:10:34.752781 27793 ts_tablet_manager.cc:1939] T b8c14195a0f5442caa8104734d0348b4 P b67c9524c1d749cdbce11cd22752eb7e: Deleting consensus metadata
I20260925 10:10:34.752912 27794 ts_tablet_manager.cc:1939] T b8c14195a0f5442caa8104734d0348b4 P 89faffa2bb504b568f49fb1e19fc34ad: Deleting consensus metadata
I20260925 10:10:34.753441 27257 catalog_manager.cc:4841] TS 89faffa2bb504b568f49fb1e19fc34ad (127.14.41.66:41239): tablet b8c14195a0f5442caa8104734d0348b4 (table test-table [id=0788976e47f94c3aac3bbb3bdc119a5f]) successfully deleted
I20260925 10:10:34.754040 27254 catalog_manager.cc:4841] TS b67c9524c1d749cdbce11cd22752eb7e (127.14.41.67:42351): tablet b8c14195a0f5442caa8104734d0348b4 (table test-table [id=0788976e47f94c3aac3bbb3bdc119a5f]) successfully deleted
W20260925 10:10:34.760557 27257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.41.65:42813 (error 108)
I20260925 10:10:34.768146 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 27472
I20260925 10:10:34.769236 27764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:34.768257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:42813 (local address 127.0.0.1:58276)
0925 10:10:34.769119 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:34.769184 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.65:42813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 10:10:34.769399 27257 catalog_manager.cc:4584] TS a91da2248cd249338c53668a08276a6c (127.14.41.65:42813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b8c14195a0f5442caa8104734d0348b4: Network error: Client connection negotiation failed: client connection to 127.14.41.65:42813: connect: Connection refused (error 111)
I20260925 10:10:34.774577 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 27605
I20260925 10:10:34.782734 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 27237
I20260925 10:10:34.789278 14501 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 10:10:35.134395 14501 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 10:10:35.134510 14501 mini_ranger.cc:66] Stopping Ranger...
I20260925 10:10:35.479692 14501 mini_ranger.cc:68] Stopped Ranger
2026-09-25 10:10:35.479 UTC [26304] LOG: received smart shutdown request
2026-09-25 10:10:35.486 UTC [26304] LOG: background worker "logical replication launcher" (PID 26313) exited with exit code 1
2026-09-25 10:10:35.486 UTC [26308] LOG: shutting down
2026-09-25 10:10:35.667 UTC [26304] LOG: database system is shut down
2026-09-25T10:10:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48381 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:10:35 dist-test-slave-s6tm krb5kdc[27804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:10:35 dist-test-slave-s6tm krb5kdc[27804](info): set up 2 sockets
Sep 25 10:10:35 dist-test-slave-s6tm krb5kdc[27804](info): commencing operation
krb5kdc: starting...
W20260925 10:10:36.897323 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:10:36 dist-test-slave-s6tm krb5kdc[27804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:10:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:10:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:10:37.024848 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:45003
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:45003,127.14.41.125:34319,127.14.41.124:38287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:10:37.107358 27817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:37.107626 27817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:37.107666 27817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:37.108847 27817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:10:37.108923 27817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:37.108948 27817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:37.108969 27817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:10:37.109000 27817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:10:37.110566 27817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:45003,127.14.41.125:34319,127.14.41.124:38287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:45003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:37.110855 27817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:37.111091 27817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:37.116333 27825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:37.116369 27823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:37.116369 27817 server_base.cc:1034] running on GCE node
W20260925 10:10:37.116340 27822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:37.116823 27817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:37.117159 27817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:37.118352 27817 hybrid_clock.cc:648] HybridClock initialized: now 1790331037118338 us; error 39 us; skew 500 ppm
Sep 25 10:10:37 dist-test-slave-s6tm krb5kdc[27804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331037, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:37.127007 27817 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:10:37.127487 27817 webserver.cc:460] Webserver started at http://127.14.41.126:35657/ using document root <none> and password file <none>
I20260925 10:10:37.127772 27817 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:37.127846 27817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:37.128010 27817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:10:37.129184 27817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "dbf07cc71be14cf68d0d346c8dafb5d8"
format_stamp: "Formatted at 2026-09-25 10:10:37 on dist-test-slave-s6tm"
server_key: "eaa73e667fa07bf18bcd0955cc3fbc1a"
server_key_iv: "472b556546a3267dd0458b87827d7ab3"
server_key_version: "encryptionkey@0"
I20260925 10:10:37.129570 27817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "dbf07cc71be14cf68d0d346c8dafb5d8"
format_stamp: "Formatted at 2026-09-25 10:10:37 on dist-test-slave-s6tm"
server_key: "eaa73e667fa07bf18bcd0955cc3fbc1a"
server_key_iv: "472b556546a3267dd0458b87827d7ab3"
server_key_version: "encryptionkey@0"
I20260925 10:10:37.130908 27817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:10:37.131711 27832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:37.131909 27817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:10:37.132046 27817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "dbf07cc71be14cf68d0d346c8dafb5d8"
format_stamp: "Formatted at 2026-09-25 10:10:37 on dist-test-slave-s6tm"
server_key: "eaa73e667fa07bf18bcd0955cc3fbc1a"
server_key_iv: "472b556546a3267dd0458b87827d7ab3"
server_key_version: "encryptionkey@0"
I20260925 10:10:37.132122 27817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:37.151448 27817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:37.153601 27817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:37.153777 27817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:37.157949 27817 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:45003
I20260925 10:10:37.157987 27884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:45003 every 8 connection(s)
I20260925 10:10:37.158370 27817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:10:37.159307 27885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:10:37.160197 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 27817
I20260925 10:10:37.160321 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:10:37.160514 14501 external_mini_cluster.cc:1428] Setting key c08d144c558a51dba1e7237fe6159630
I20260925 10:10:37.161242 27885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.41.125@KRBTEST.COM; defaulting to no policy
Sep 25 10:10:37 dist-test-slave-s6tm krb5kdc[27804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331037, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.125@KRBTEST.COM" created.
I20260925 10:10:37.171391 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.162100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40464 (local address 127.14.41.126:45003)
0925 10:10:37.162295 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:37.162306 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:37.162411 (+ 105us) server_negotiation.cc:407] Connection header received
0925 10:10:37.162956 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:37.162959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:37.162984 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:37.163082 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:37.163653 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.164573 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.165466 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.165801 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.165936 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.168135 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:37.168144 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:37.168144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:37.168151 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:37.169624 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.170060 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.170063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.170063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.170142 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.170393 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.170393 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.170394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.170754 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:37.170841 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:37.171113 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 10:10:37.171216 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":94,"threads_started":1}
I20260925 10:10:37.171474 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.162096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:45003 (local address 127.0.0.1:40464)
0925 10:10:37.162270 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.162309 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:37.162780 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:37.163087 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:37.163089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:37.163373 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:37.163529 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:37.163534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.164669 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.164670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:37.165366 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:37.165368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.165897 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.165898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:37.165968 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.166618 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:37.166627 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:37.168042 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:37.169711 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:37.169718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:37.169720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:37.169982 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:37.170202 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:37.170203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:37.170203 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:37.170322 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:37.170853 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:37.170861 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:37.170961 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 10:10:37.171074 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 10:10:37.171327 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":68,"thread_start_us":92,"threads_started":1}
I20260925 10:10:37.172288 27885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:10:37.172701 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.172558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:34319 (local address 127.0.0.1:47782)
0925 10:10:37.172616 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.172672 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.125:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 10:10:37.172853 27833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.41.125:34319: connect: Connection refused (error 111)
I20260925 10:10:37.173290 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.173190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:34319 (local address 127.0.0.1:47798)
0925 10:10:37.173234 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.173264 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.125:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 10:10:37.173439 27885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.125:34319: Network error: Client connection negotiation failed: client connection to 127.14.41.125:34319: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.125.keytab.
Entry for principal kudu/127.14.41.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.125.keytab.
I20260925 10:10:37.187132 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.125:34319
--webserver_interface=127.14.41.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.125.keytab
--principal=kudu/127.14.41.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:45003,127.14.41.125:34319,127.14.41.124:38287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:10:37.222726 27885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } attempt: 1
I20260925 10:10:37.223374 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.223189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:34319 (local address 127.0.0.1:47804)
0925 10:10:37.223283 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.223338 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.125:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 10:10:37.223840 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.223705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:34319 (local address 127.0.0.1:47810)
0925 10:10:37.223778 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.223814 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.125:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 10:10:37.224047 27885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.125:34319: Network error: Client connection negotiation failed: client connection to 127.14.41.125:34319: connect: Connection refused (error 111)
W20260925 10:10:37.270519 27892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:37.270741 27892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:37.270763 27892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:37.271967 27892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:10:37.272011 27892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:37.272025 27892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:37.272038 27892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:10:37.272050 27892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:10:37.274528 27892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:45003,127.14.41.125:34319,127.14.41.124:38287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.125.keytab
--principal=kudu/127.14.41.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.125:34319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.41.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:37.274874 27892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:37.275121 27892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:37.280292 27898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:37.280292 27897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:37.280306 27900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:37.280361 27892 server_base.cc:1034] running on GCE node
I20260925 10:10:37.280915 27892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:37.281220 27892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:37.282378 27892 hybrid_clock.cc:648] HybridClock initialized: now 1790331037282369 us; error 35 us; skew 500 ppm
Sep 25 10:10:37 dist-test-slave-s6tm krb5kdc[27804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331037, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:37.285000 27892 init.cc:377] Logged in from keytab as kudu/127.14.41.125@KRBTEST.COM (short username kudu)
I20260925 10:10:37.285398 27892 webserver.cc:460] Webserver started at http://127.14.41.125:40227/ using document root <none> and password file <none>
I20260925 10:10:37.285619 27892 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:37.285665 27892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:37.285765 27892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:10:37.286809 27892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-1/data/instance:
uuid: "dd2318a780904536a01c51db729ba1b7"
format_stamp: "Formatted at 2026-09-25 10:10:37 on dist-test-slave-s6tm"
server_key: "1953940e92058a45be9d72bff529a538"
server_key_iv: "9ec80a36828528876d6bc1bceb683e98"
server_key_version: "encryptionkey@0"
I20260925 10:10:37.287127 27892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-1/wal/instance:
uuid: "dd2318a780904536a01c51db729ba1b7"
format_stamp: "Formatted at 2026-09-25 10:10:37 on dist-test-slave-s6tm"
server_key: "1953940e92058a45be9d72bff529a538"
server_key_iv: "9ec80a36828528876d6bc1bceb683e98"
server_key_version: "encryptionkey@0"
I20260925 10:10:37.288493 27892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:10:37.289211 27907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:37.289400 27892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:10:37.289491 27892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-1/wal
uuid: "dd2318a780904536a01c51db729ba1b7"
format_stamp: "Formatted at 2026-09-25 10:10:37 on dist-test-slave-s6tm"
server_key: "1953940e92058a45be9d72bff529a538"
server_key_iv: "9ec80a36828528876d6bc1bceb683e98"
server_key_version: "encryptionkey@0"
I20260925 10:10:37.289552 27892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:37.292184 27885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } attempt: 2
I20260925 10:10:37.292588 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.292470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:34319 (local address 127.0.0.1:47812)
0925 10:10:37.292523 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.292559 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.125:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 10:10:37.292950 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.292885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:34319 (local address 127.0.0.1:47826)
0925 10:10:37.292917 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.292938 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.125:34319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 10:10:37.293048 27885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.125:34319: Network error: Client connection negotiation failed: client connection to 127.14.41.125:34319: connect: Connection refused (error 111)
I20260925 10:10:37.296860 27892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:37.298940 27892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:37.299098 27892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:37.303738 27892 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.125:34319
I20260925 10:10:37.303793 27959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.125:34319 every 8 connection(s)
I20260925 10:10:37.304224 27892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-1/data/info.pb
I20260925 10:10:37.305158 27960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:10:37.306742 27960 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:10:37.312500 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 27892
I20260925 10:10:37.312652 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-1/wal/instance
Sep 25 10:10:37 dist-test-slave-s6tm krb5kdc[27804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331037, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.125@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:10:37.312842 14501 external_mini_cluster.cc:1428] Setting key 3379be24b82fa06f94b75895df038f12
I20260925 10:10:37.316597 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.307506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40470 (local address 127.14.41.126:45003)
0925 10:10:37.307589 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:37.307591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:37.307751 (+ 160us) server_negotiation.cc:407] Connection header received
0925 10:10:37.308262 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:37.308264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:37.308278 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:37.308347 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:37.308948 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.309475 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.310584 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.310800 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.310864 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.313320 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:37.313321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:37.313322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:37.313330 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:37.314908 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.315457 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.315459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.315459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.315513 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.315822 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.315823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.315823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.315972 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:37.316032 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:37.316399 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 10:10:37.316492 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:10:37.316888 27962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.307434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:45003 (local address 127.0.0.1:40470)
0925 10:10:37.307623 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.307666 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:37.308169 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:37.308403 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:37.308407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:37.308668 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:37.308867 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:37.308871 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.309593 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.309594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:37.310491 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:37.310494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.310876 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.310877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:37.310983 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.311589 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:37.311598 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:37.313219 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:37.315025 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:37.315034 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:37.315036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:37.315365 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:37.315601 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:37.315602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:37.315602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:37.315758 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:37.316034 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:37.316041 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:37.316220 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 10:10:37.316382 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 10:10:37.316583 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":91,"threads_started":1}
I20260925 10:10:37.317534 27960 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.41.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.124@KRBTEST.COM" created.
Sep 25 10:10:37 dist-test-slave-s6tm krb5kdc[27804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331037, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.125@KRBTEST.COM for kudu/127.14.41.125@KRBTEST.COM
I20260925 10:10:37.327749 27962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.317832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:34319 (local address 127.0.0.1:47838)
0925 10:10:37.317921 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.317932 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:37.317973 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:37.318289 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:37.318290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:37.318406 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:37.318456 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:37.318457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.319794 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.319795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:37.320359 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:37.320361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.321039 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.321039 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:37.321104 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.321609 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:37.321614 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:37.324737 (+ 3123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:37.326650 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:37.326651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:37.326651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:37.326838 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:37.327088 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:37.327089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:37.327089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:37.327142 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:37.327478 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:37.327479 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:37.327514 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 10:10:37.327569 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 10:10:37.327655 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 10:10:37.327956 27964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.317929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47838 (local address 127.14.41.125:34319)
0925 10:10:37.318109 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:37.318115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:37.318142 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:10:37.318169 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:37.318171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:37.318188 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:37.318280 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:37.318529 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.319701 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.320439 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.320948 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.321042 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.324832 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:37.324843 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:37.324843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:37.324851 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:37.326564 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.326925 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.326929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.326929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.326994 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.327201 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.327202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.327203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.327412 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:37.327500 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:37.327603 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 10:10:37.327752 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260925 10:10:37.328650 27960 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:10:37.329034 27962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.328880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:38287 (local address 127.0.0.1:34078)
0925 10:10:37.328932 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.328991 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:38287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 10:10:37.329183 27908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.41.124:38287: connect: Connection refused (error 111)
I20260925 10:10:37.329625 27962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.329515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:38287 (local address 127.0.0.1:34090)
0925 10:10:37.329563 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.329594 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:38287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 10:10:37.329746 27960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.124:38287: Network error: Client connection negotiation failed: client connection to 127.14.41.124:38287: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.124.keytab.
Entry for principal kudu/127.14.41.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.124.keytab.
I20260925 10:10:37.339294 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.124:38287
--webserver_interface=127.14.41.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.124.keytab
--principal=kudu/127.14.41.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:45003,127.14.41.125:34319,127.14.41.124:38287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:10:37.379027 27960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } attempt: 1
I20260925 10:10:37.379647 27962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.379464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:38287 (local address 127.0.0.1:34096)
0925 10:10:37.379563 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.379611 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:38287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 10:10:37.380079 27962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.379993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:38287 (local address 127.0.0.1:34102)
0925 10:10:37.380028 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.380055 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:38287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 10:10:37.380192 27960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.124:38287: Network error: Client connection negotiation failed: client connection to 127.14.41.124:38287: connect: Connection refused (error 111)
I20260925 10:10:37.384214 27885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } attempt: 3
Sep 25 10:10:37 dist-test-slave-s6tm krb5kdc[27804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331037, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for kudu/127.14.41.125@KRBTEST.COM
I20260925 10:10:37.391724 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.384510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:34319 (local address 127.0.0.1:47844)
0925 10:10:37.384569 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.384581 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:37.384636 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:37.384791 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:37.384792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:37.384945 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:37.385015 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:37.385017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.385977 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.385978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:37.386518 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:37.386519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.386997 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.386998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:37.387071 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.387581 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:37.387587 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:37.389028 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:37.390806 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:37.390807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:37.390807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:37.390971 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:37.391165 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:37.391166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:37.391166 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:37.391218 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:37.391479 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:37.391480 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:37.391518 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 10:10:37.391572 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 10:10:37.391651 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 10:10:37.391782 27964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.384581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47844 (local address 127.14.41.125:34319)
0925 10:10:37.384623 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:37.384625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:37.384637 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:10:37.384709 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:37.384710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:37.384721 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:37.384777 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:37.385118 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.385887 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.386595 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.386918 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.386985 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.389132 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:37.389134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:37.389135 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:37.389144 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:37.390713 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.391038 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.391039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.391040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.391102 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.391274 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.391275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.391275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.391417 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:37.391485 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:37.391614 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 10:10:37.391693 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:10:37.392259 27885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:10:37.392614 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.392509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:38287 (local address 127.0.0.1:34110)
0925 10:10:37.392555 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.392585 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:38287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 10:10:37.392937 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.392863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:38287 (local address 127.0.0.1:34120)
0925 10:10:37.392901 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.392923 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:38287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 10:10:37.393046 27885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.124:38287: Network error: Client connection negotiation failed: client connection to 127.14.41.124:38287: connect: Connection refused (error 111)
I20260925 10:10:37.424216 27885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } attempt: 1
I20260925 10:10:37.424751 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.424618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:38287 (local address 127.0.0.1:34124)
0925 10:10:37.424679 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.424718 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:38287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 10:10:37.425169 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.425082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:38287 (local address 127.0.0.1:34134)
0925 10:10:37.425127 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.425153 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:38287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 10:10:37.425274 27885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.124:38287: Network error: Client connection negotiation failed: client connection to 127.14.41.124:38287: connect: Connection refused (error 111)
W20260925 10:10:37.425577 27967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:37.425776 27967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:37.425830 27967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:37.427026 27967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:10:37.427071 27967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:37.427086 27967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:37.427134 27967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:10:37.427152 27967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:10:37.428759 27967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:45003,127.14.41.125:34319,127.14.41.124:38287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.124.keytab
--principal=kudu/127.14.41.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.124:38287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.41.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:37.429019 27967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:37.429251 27967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:10:37.434445 27967 server_base.cc:1034] running on GCE node
W20260925 10:10:37.434346 27972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:37.434337 27973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:37.434389 27975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:37.434899 27967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:37.435178 27967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:37.436314 27967 hybrid_clock.cc:648] HybridClock initialized: now 1790331037436299 us; error 37 us; skew 500 ppm
Sep 25 10:10:37 dist-test-slave-s6tm krb5kdc[27804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331037, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:37.438735 27967 init.cc:377] Logged in from keytab as kudu/127.14.41.124@KRBTEST.COM (short username kudu)
I20260925 10:10:37.439194 27967 webserver.cc:460] Webserver started at http://127.14.41.124:39591/ using document root <none> and password file <none>
I20260925 10:10:37.439393 27967 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:37.439435 27967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:37.439532 27967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:10:37.440613 27967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-2/data/instance:
uuid: "80918c46c344407cb574e46a24e4cf58"
format_stamp: "Formatted at 2026-09-25 10:10:37 on dist-test-slave-s6tm"
server_key: "0f5c52516300fab4893c9413fa0105a0"
server_key_iv: "52dd812bc78a1ed3c0caf4601857036d"
server_key_version: "encryptionkey@0"
I20260925 10:10:37.440920 27967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-2/wal/instance:
uuid: "80918c46c344407cb574e46a24e4cf58"
format_stamp: "Formatted at 2026-09-25 10:10:37 on dist-test-slave-s6tm"
server_key: "0f5c52516300fab4893c9413fa0105a0"
server_key_iv: "52dd812bc78a1ed3c0caf4601857036d"
server_key_version: "encryptionkey@0"
I20260925 10:10:37.442175 27967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:10:37.442862 27982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:37.443025 27967 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:10:37.443125 27967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-2/wal
uuid: "80918c46c344407cb574e46a24e4cf58"
format_stamp: "Formatted at 2026-09-25 10:10:37 on dist-test-slave-s6tm"
server_key: "0f5c52516300fab4893c9413fa0105a0"
server_key_iv: "52dd812bc78a1ed3c0caf4601857036d"
server_key_version: "encryptionkey@0"
I20260925 10:10:37.443199 27967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:37.448349 27960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } attempt: 2
I20260925 10:10:37.448793 27962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.448670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:38287 (local address 127.0.0.1:34144)
0925 10:10:37.448727 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.448763 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:38287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 10:10:37.449177 27962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.449090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:38287 (local address 127.0.0.1:34146)
0925 10:10:37.449130 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.449159 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:38287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 10:10:37.449276 27960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.124:38287: Network error: Client connection negotiation failed: client connection to 127.14.41.124:38287: connect: Connection refused (error 111)
I20260925 10:10:37.461184 27967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:37.463605 27967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:37.463733 27967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:37.467550 27967 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.124:38287
I20260925 10:10:37.467614 28034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.124:38287 every 8 connection(s)
I20260925 10:10:37.468022 27967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-2/data/info.pb
I20260925 10:10:37.468943 28035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:10:37.470587 28035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:10:37.474380 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 27967
I20260925 10:10:37.474479 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790330877480195-14501-0/minicluster-data/master-2/wal/instance
I20260925 10:10:37.474720 14501 external_mini_cluster.cc:1428] Setting key 2576787b492ad09ea316be39d02b2f8a
Sep 25 10:10:37 dist-test-slave-s6tm krb5kdc[27804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331037, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.124@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
Sep 25 10:10:37 dist-test-slave-s6tm krb5kdc[27804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.124@KRBTEST.COM
I20260925 10:10:37.483062 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.471393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40478 (local address 127.14.41.126:45003)
0925 10:10:37.471654 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:37.471656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:37.471805 (+ 149us) server_negotiation.cc:407] Connection header received
0925 10:10:37.472339 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:37.472341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:37.472352 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:37.472417 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:37.473133 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.473767 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.475071 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.475401 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.475475 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.478587 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:37.478588 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:37.478589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:37.478607 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:37.481353 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.481929 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.481931 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.481932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.482004 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.482322 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.482323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.482323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.482465 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:37.482541 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:37.482843 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 10:10:37.482940 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260925 10:10:37.483323 28037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.471371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:45003 (local address 127.0.0.1:40478)
0925 10:10:37.471659 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.471703 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:37.472239 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:37.472475 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:37.472478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:37.472741 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:37.473007 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:37.473013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.473867 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.473868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:37.474927 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:37.474931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.475480 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.475481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:37.475641 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.476701 (+ 1060us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:37.476712 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:37.478493 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:37.481456 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:37.481466 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:37.481468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:37.481810 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:37.482095 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:37.482097 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:37.482097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:37.482256 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:37.482543 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:37.482550 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:37.482705 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 10:10:37.482821 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 10:10:37.483058 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":99,"threads_started":1}
I20260925 10:10:37.483899 28035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:10:37.484876 28039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.475148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34148 (local address 127.14.41.124:38287)
0925 10:10:37.475327 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:37.475332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:37.475399 (+ 67us) server_negotiation.cc:407] Connection header received
0925 10:10:37.475471 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:37.475474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:37.475497 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:37.475580 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:37.476227 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.477305 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.478365 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.478820 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.478895 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.481762 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:37.481768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:37.481768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:37.481776 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:37.483578 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.483921 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.483944 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.483945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.484003 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.484267 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.484268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.484269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.484429 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:37.484585 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:37.484680 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 10:10:37.484762 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":99,"threads_started":1}
Sep 25 10:10:37 dist-test-slave-s6tm krb5kdc[27804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331037, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.124@KRBTEST.COM for kudu/127.14.41.125@KRBTEST.COM
Sep 25 10:10:37 dist-test-slave-s6tm krb5kdc[27804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:10:37.491276 27964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.484274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.14.41.125:34319)
0925 10:10:37.484327 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:37.484329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:37.484339 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:10:37.484364 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:37.484365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:37.484374 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:37.484429 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:37.484682 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.485350 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.486391 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.486621 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.486713 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.488538 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:37.488539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:37.488540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:37.488548 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:37.489918 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.490302 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.490303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.490303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.490350 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.490554 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.490555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.490555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.490692 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:37.490773 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:37.491091 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 10:10:37.491176 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:10:37.491289 28037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.484196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:34319 (local address 127.0.0.1:47850)
0925 10:10:37.484244 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.484255 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:37.484291 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:37.484432 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:37.484433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:37.484571 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:37.484605 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:37.484607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.485426 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.485427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:37.486299 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:37.486302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.486707 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.486708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:37.486785 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.487219 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:37.487224 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:37.488468 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:37.489996 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:37.489997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:37.489998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:37.490225 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:37.490415 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:37.490416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:37.490416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:37.490487 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:37.490947 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:37.490948 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:37.490985 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 10:10:37.491041 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 10:10:37.491158 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 10:10:37.491721 28035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:10:37.493394 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.485719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40494 (local address 127.14.41.126:45003)
0925 10:10:37.485755 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:37.485757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:37.485765 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:10:37.485832 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:37.485832 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:37.485839 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:37.485878 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:37.486421 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.486865 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.487516 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.487786 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.487856 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.490535 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:37.490537 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:37.490538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:37.490545 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:37.492266 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.492620 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.492621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.492622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.492675 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.492875 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.492875 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.492876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.492993 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:37.493185 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:37.493221 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 10:10:37.493293 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 25 10:10:37 dist-test-slave-s6tm krb5kdc[27804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331037, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.124@KRBTEST.COM for kudu/127.14.41.124@KRBTEST.COM
Sep 25 10:10:37 dist-test-slave-s6tm krb5kdc[27804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.125@KRBTEST.COM
I20260925 10:10:37.499434 28039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.492121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34158 (local address 127.14.41.124:38287)
0925 10:10:37.492155 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:37.492156 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:37.492180 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:10:37.492249 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:37.492250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:37.492260 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:37.492314 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:37.492646 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.493208 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.494179 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.494389 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.494469 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.496187 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:37.496189 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:37.496189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:37.496196 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:37.497865 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.498246 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.498248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.498248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.498308 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.498675 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.498676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.498676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.498826 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:37.498901 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:37.499164 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 10:10:37.499289 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":18}
I20260925 10:10:37.499449 28037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.492084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:38287 (local address 127.0.0.1:34158)
0925 10:10:37.492133 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.492145 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:37.492195 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:37.492317 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:37.492318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:37.492537 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:37.492569 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:37.492571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.493277 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.493278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:37.494086 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:37.494090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.494466 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.494467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:37.494514 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.494910 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:37.494915 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:37.496096 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:37.497975 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:37.497977 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:37.497977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:37.498168 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:37.498525 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:37.498526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:37.498526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:37.498598 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:37.498961 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:37.498963 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:37.499020 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 10:10:37.499114 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 10:10:37.499240 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 10:10:37.500402 27885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } attempt: 2
I20260925 10:10:37.501915 28035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58: Bootstrap starting.
I20260925 10:10:37.502243 27964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.494026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.14.41.125:34319)
0925 10:10:37.494063 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:37.494065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:37.494074 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:10:37.494138 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:37.494139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:37.494148 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:37.494191 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:37.494645 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.495113 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.495750 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.496062 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.496133 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.498530 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:37.498531 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:37.498531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:37.498539 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:37.500373 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.500806 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.500808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.500808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.500872 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.501251 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.501252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.501253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.501423 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:37.501625 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:37.502050 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 10:10:37.502134 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:10:37.503259 28035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58: Neither blocks nor log segments found. Creating new log.
I20260925 10:10:37.503592 28035 log.cc:824] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58: Log is configured to *not* fsync() on all Append() calls
Sep 25 10:10:37 dist-test-slave-s6tm krb5kdc[27804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331037, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for kudu/127.14.41.124@KRBTEST.COM
I20260925 10:10:37.504699 28035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58: No bootstrap required, opened a new log
I20260925 10:10:37.506819 28035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } } peers { permanent_uuid: "dd2318a780904536a01c51db729ba1b7" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } } peers { permanent_uuid: "80918c46c344407cb574e46a24e4cf58" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } }
I20260925 10:10:37.507000 28035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:10:37.507071 28035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80918c46c344407cb574e46a24e4cf58, State: Initialized, Role: FOLLOWER
I20260925 10:10:37.507256 28035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } } peers { permanent_uuid: "dd2318a780904536a01c51db729ba1b7" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } } peers { permanent_uuid: "80918c46c344407cb574e46a24e4cf58" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } }
I20260925 10:10:37.507513 28039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.500898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34162 (local address 127.14.41.124:38287)
0925 10:10:37.500954 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:37.500955 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:37.500962 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:10:37.500980 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:37.500981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:37.500989 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:37.501065 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:37.501382 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.501835 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.502626 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.502830 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.502911 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.504944 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:37.504946 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:37.504946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:37.504954 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:37.506336 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.506698 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.506699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.506700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.506756 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.506957 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.506958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.506958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.507127 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:37.507194 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:37.507344 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 10:10:37.507422 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 10:10:37.507514 27888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.500727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:38287 (local address 127.0.0.1:34162)
0925 10:10:37.500783 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.500794 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:37.500829 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:37.501082 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:37.501083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:37.501259 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:37.501307 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:37.501310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.501912 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.501913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:37.502543 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:37.502544 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.502911 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.502912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:37.502969 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.503423 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:37.503428 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:37.504859 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:37.506437 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:37.506439 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:37.506439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:37.506617 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:37.506829 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:37.506830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:37.506831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:37.506893 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:37.507186 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:37.507187 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:37.507234 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 10:10:37.507319 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 10:10:37.507422 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 10:10:37.507601 28035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:10:37.508015 28041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } } peers { permanent_uuid: "dd2318a780904536a01c51db729ba1b7" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } } peers { permanent_uuid: "80918c46c344407cb574e46a24e4cf58" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } } }
I20260925 10:10:37.508191 28041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:10:37.509210 27885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8: Bootstrap starting.
I20260925 10:10:37.509971 27885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8: Neither blocks nor log segments found. Creating new log.
W20260925 10:10:37.510072 28052 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:10:37.510181 28052 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:10:37.510319 27885 log.cc:824] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8: Log is configured to *not* fsync() on all Append() calls
I20260925 10:10:37.511732 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:10:37.512609 27885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8: No bootstrap required, opened a new log
I20260925 10:10:37.510239 27967 master_runner.cc:186] Error getting master registration for 127.14.41.126:45003: OK, instance_id { permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" instance_seqno: 1790331037156793 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:10:37.513453 27967 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:10:37.514274 27885 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } } peers { permanent_uuid: "dd2318a780904536a01c51db729ba1b7" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } } peers { permanent_uuid: "80918c46c344407cb574e46a24e4cf58" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } }
I20260925 10:10:37.514370 27885 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:10:37.514391 27885 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbf07cc71be14cf68d0d346c8dafb5d8, State: Initialized, Role: FOLLOWER
I20260925 10:10:37.514510 27885 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } } peers { permanent_uuid: "dd2318a780904536a01c51db729ba1b7" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } } peers { permanent_uuid: "80918c46c344407cb574e46a24e4cf58" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } }
I20260925 10:10:37.514761 27885 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:10:37.514791 28059 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } } peers { permanent_uuid: "dd2318a780904536a01c51db729ba1b7" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } } peers { permanent_uuid: "80918c46c344407cb574e46a24e4cf58" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } } }
I20260925 10:10:37.514887 28059 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:10:37.516438 28070 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:10:37.516724 28070 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:10:37.517478 27817 master_runner.cc:186] Error getting master registration for 127.14.41.125:34319: OK, instance_id { permanent_uuid: "dd2318a780904536a01c51db729ba1b7" instance_seqno: 1790331037302533 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:10:37.520915 27817 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:10:37.530890 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.521697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40498 (local address 127.14.41.126:45003)
0925 10:10:37.521741 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:37.521743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:37.522173 (+ 430us) server_negotiation.cc:407] Connection header received
0925 10:10:37.522261 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:37.522262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:37.522272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:37.522304 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:37.523119 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.523866 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.525032 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.525320 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.525364 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.527293 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:37.527294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:37.527294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:37.527302 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:37.529307 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.530035 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.530036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.530037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.530098 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.530297 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.530297 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.530298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.530422 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:37.530494 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:37.530710 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 10:10:37.530789 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:10:37.532681 27964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.521226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47862 (local address 127.14.41.125:34319)
0925 10:10:37.521277 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:37.521280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:37.521894 (+ 614us) server_negotiation.cc:407] Connection header received
0925 10:10:37.521954 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:37.521956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:37.521969 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:37.522043 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:37.523808 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.524652 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.525817 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.526152 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.526196 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.528693 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:37.528694 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:37.528694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:37.528703 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:37.530563 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.531049 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.531050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.531051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.531120 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.532028 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.532029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.532030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.532173 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:37.532261 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:37.532491 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 10:10:37.532585 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:10:37.538000 28039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.521520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34174 (local address 127.14.41.124:38287)
0925 10:10:37.521584 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:37.521586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:37.522091 (+ 505us) server_negotiation.cc:407] Connection header received
0925 10:10:37.522119 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:37.522120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:37.522133 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:37.522194 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:37.526626 (+ 4432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.527453 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.532131 (+ 4678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.532509 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.532556 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.534561 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:37.534563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:37.534563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:37.534572 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:37.536539 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.537036 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.537037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.537038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.537104 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.537326 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.537327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.537327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.537479 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:37.537555 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:37.537797 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 10:10:37.537888 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:10:37.540431 27960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } attempt: 3
Sep 25 10:10:37 dist-test-slave-s6tm krb5kdc[27804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331037, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.125@KRBTEST.COM for kudu/127.14.41.124@KRBTEST.COM
I20260925 10:10:37.549170 27962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.540773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:38287 (local address 127.0.0.1:34184)
0925 10:10:37.540836 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:37.540849 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:37.540901 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:37.541049 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:37.541050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:37.541247 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:37.541299 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:37.541301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.542265 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.542266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:37.543077 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:37.543080 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.543634 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.543635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:37.543699 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.544276 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:37.544282 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:37.545872 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:37.547975 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:37.547977 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:37.547977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:37.548197 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:37.548427 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:37.548428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:37.548428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:37.548497 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:37.548827 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:37.548828 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:37.548874 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 10:10:37.548940 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 10:10:37.549050 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 10:10:37.549511 28039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:37.540807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34184 (local address 127.14.41.124:38287)
0925 10:10:37.540891 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:37.540893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:37.540903 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:10:37.540968 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:37.540969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:37.540980 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:37.541037 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:37.541399 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.542163 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.543174 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:37.543554 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:37.543635 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:37.545949 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:37.545951 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:37.545951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:37.545960 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:37.547846 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.548278 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.548279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.548279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.548349 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:37.548589 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:37.548590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:37.548591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:37.548738 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:37.548801 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:37.549340 (+ 539us) server_negotiation.cc:299] Negotiation successful
0925 10:10:37.549422 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:10:37.551529 27960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7: Bootstrap starting.
I20260925 10:10:37.552376 27960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7: Neither blocks nor log segments found. Creating new log.
I20260925 10:10:37.552718 27960 log.cc:824] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7: Log is configured to *not* fsync() on all Append() calls
I20260925 10:10:37.553676 27960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7: No bootstrap required, opened a new log
I20260925 10:10:37.555763 27960 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } } peers { permanent_uuid: "dd2318a780904536a01c51db729ba1b7" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } } peers { permanent_uuid: "80918c46c344407cb574e46a24e4cf58" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } }
I20260925 10:10:37.555984 27960 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:10:37.556051 27960 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd2318a780904536a01c51db729ba1b7, State: Initialized, Role: FOLLOWER
I20260925 10:10:37.556211 27960 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } } peers { permanent_uuid: "dd2318a780904536a01c51db729ba1b7" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } } peers { permanent_uuid: "80918c46c344407cb574e46a24e4cf58" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } }
I20260925 10:10:37.556593 27960 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:10:37.556572 28076 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } } peers { permanent_uuid: "dd2318a780904536a01c51db729ba1b7" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } } peers { permanent_uuid: "80918c46c344407cb574e46a24e4cf58" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } } }
I20260925 10:10:37.556788 28076 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:10:37.558784 28087 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:10:37.558851 28087 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:10:37.561021 27892 master_runner.cc:263] No leader master found from master dd2318a780904536a01c51db729ba1b7
I20260925 10:10:37.561106 27892 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:10:37.608537 28076 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:10:37.608701 28076 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } } peers { permanent_uuid: "dd2318a780904536a01c51db729ba1b7" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } } peers { permanent_uuid: "80918c46c344407cb574e46a24e4cf58" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } }
I20260925 10:10:37.608973 28076 leader_election.cc:288] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbf07cc71be14cf68d0d346c8dafb5d8 (127.14.41.126:45003), 80918c46c344407cb574e46a24e4cf58 (127.14.41.124:38287)
I20260925 10:10:37.609345 28009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd2318a780904536a01c51db729ba1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80918c46c344407cb574e46a24e4cf58" is_pre_election: true
I20260925 10:10:37.609372 27859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd2318a780904536a01c51db729ba1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" is_pre_election: true
I20260925 10:10:37.609539 28009 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd2318a780904536a01c51db729ba1b7 in term 0.
I20260925 10:10:37.609547 27859 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd2318a780904536a01c51db729ba1b7 in term 0.
I20260925 10:10:37.609757 27908 leader_election.cc:302] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80918c46c344407cb574e46a24e4cf58, dd2318a780904536a01c51db729ba1b7; no voters:
I20260925 10:10:37.609905 28076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:10:37.609951 28076 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:10:37.609973 28076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:10:37.610978 28076 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } } peers { permanent_uuid: "dd2318a780904536a01c51db729ba1b7" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } } peers { permanent_uuid: "80918c46c344407cb574e46a24e4cf58" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } }
I20260925 10:10:37.611129 28076 leader_election.cc:288] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [CANDIDATE]: Term 1 election: Requested vote from peers dbf07cc71be14cf68d0d346c8dafb5d8 (127.14.41.126:45003), 80918c46c344407cb574e46a24e4cf58 (127.14.41.124:38287)
I20260925 10:10:37.611317 28009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd2318a780904536a01c51db729ba1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80918c46c344407cb574e46a24e4cf58"
I20260925 10:10:37.611385 27859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd2318a780904536a01c51db729ba1b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8"
I20260925 10:10:37.611425 28009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:10:37.611474 27859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:10:37.612493 27859 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd2318a780904536a01c51db729ba1b7 in term 1.
I20260925 10:10:37.612504 28009 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd2318a780904536a01c51db729ba1b7 in term 1.
I20260925 10:10:37.612691 27911 leader_election.cc:302] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dbf07cc71be14cf68d0d346c8dafb5d8, dd2318a780904536a01c51db729ba1b7; no voters:
I20260925 10:10:37.612777 28076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:10:37.612950 28076 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [term 1 LEADER]: Becoming Leader. State: Replica: dd2318a780904536a01c51db729ba1b7, State: Running, Role: LEADER
I20260925 10:10:37.613075 28076 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } } peers { permanent_uuid: "dd2318a780904536a01c51db729ba1b7" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } } peers { permanent_uuid: "80918c46c344407cb574e46a24e4cf58" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } }
I20260925 10:10:37.613514 28076 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd2318a780904536a01c51db729ba1b7. Latest consensus state: current_term: 1 leader_uuid: "dd2318a780904536a01c51db729ba1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } } peers { permanent_uuid: "dd2318a780904536a01c51db729ba1b7" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } } peers { permanent_uuid: "80918c46c344407cb574e46a24e4cf58" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } } }
I20260925 10:10:37.613596 28076 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [sys.catalog]: This master's current role is: LEADER
I20260925 10:10:37.614197 28091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:10:37.614508 28091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:10:37.616205 28009 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58 [term 1 FOLLOWER]: Refusing update from remote peer dd2318a780904536a01c51db729ba1b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:10:37.616605 28076 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80918c46c344407cb574e46a24e4cf58" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:10:37.617919 28041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd2318a780904536a01c51db729ba1b7. Latest consensus state: current_term: 1 leader_uuid: "dd2318a780904536a01c51db729ba1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } } peers { permanent_uuid: "dd2318a780904536a01c51db729ba1b7" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } } peers { permanent_uuid: "80918c46c344407cb574e46a24e4cf58" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } } }
I20260925 10:10:37.618006 28041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:10:37.618461 27859 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8 [term 1 FOLLOWER]: Refusing update from remote peer dd2318a780904536a01c51db729ba1b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:10:37.618778 28076 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:10:37.620659 28059 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd2318a780904536a01c51db729ba1b7. Latest consensus state: current_term: 1 leader_uuid: "dd2318a780904536a01c51db729ba1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } } peers { permanent_uuid: "dd2318a780904536a01c51db729ba1b7" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } } peers { permanent_uuid: "80918c46c344407cb574e46a24e4cf58" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } } }
I20260925 10:10:37.620767 28059 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:10:37.625347 28076 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dd2318a780904536a01c51db729ba1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } } peers { permanent_uuid: "dd2318a780904536a01c51db729ba1b7" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } } peers { permanent_uuid: "80918c46c344407cb574e46a24e4cf58" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } } }
I20260925 10:10:37.625444 28076 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [sys.catalog]: This master's current role is: LEADER
I20260925 10:10:37.625484 28076 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dd2318a780904536a01c51db729ba1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } } peers { permanent_uuid: "dd2318a780904536a01c51db729ba1b7" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } } peers { permanent_uuid: "80918c46c344407cb574e46a24e4cf58" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } } }
I20260925 10:10:37.625523 28076 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7 [sys.catalog]: This master's current role is: LEADER
I20260925 10:10:37.626063 28041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd2318a780904536a01c51db729ba1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } } peers { permanent_uuid: "dd2318a780904536a01c51db729ba1b7" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } } peers { permanent_uuid: "80918c46c344407cb574e46a24e4cf58" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } } }
I20260925 10:10:37.626153 28041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:10:37.626693 28059 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd2318a780904536a01c51db729ba1b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbf07cc71be14cf68d0d346c8dafb5d8" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45003 } } peers { permanent_uuid: "dd2318a780904536a01c51db729ba1b7" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 34319 } } peers { permanent_uuid: "80918c46c344407cb574e46a24e4cf58" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 38287 } } }
I20260925 10:10:37.626780 28059 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:10:37.627547 28092 mvcc.cc:205] Tried to move back new op lower bound from 7333195930073800704 to 7333195930063835136. Current Snapshot: MvccSnapshot[applied={T|T < 7333195930073800704}]
I20260925 10:10:37.627789 28093 mvcc.cc:205] Tried to move back new op lower bound from 7333195930073800704 to 7333195930063835136. Current Snapshot: MvccSnapshot[applied={T|T < 7333195930073800704}]
I20260925 10:10:37.628007 28095 mvcc.cc:205] Tried to move back new op lower bound from 7333195930073800704 to 7333195930063835136. Current Snapshot: MvccSnapshot[applied={T|T < 7333195930073800704}]
I20260925 10:10:37.630148 28091 catalog_manager.cc:1347] Generated new cluster ID: c62330d33c3549c584dfea76e9a0330c
I20260925 10:10:37.630198 28091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:10:37.653244 28091 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:10:37.654428 28091 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:10:37.661448 28091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd2318a780904536a01c51db729ba1b7: Generated new TSK 0
I20260925 10:10:37.661803 28091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:10:38.510685 28052 catalog_manager.cc:1259] Loaded cluster ID: c62330d33c3549c584dfea76e9a0330c
I20260925 10:10:38.510766 28052 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58: loading cluster ID for follower catalog manager: success
I20260925 10:10:38.511613 28052 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58: acquiring CA information for follower catalog manager: success
I20260925 10:10:38.511902 28052 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 80918c46c344407cb574e46a24e4cf58: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:10:38.517168 28070 catalog_manager.cc:1259] Loaded cluster ID: c62330d33c3549c584dfea76e9a0330c
I20260925 10:10:38.517225 28070 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8: loading cluster ID for follower catalog manager: success
I20260925 10:10:38.518097 28070 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8: acquiring CA information for follower catalog manager: success
I20260925 10:10:38.518401 28070 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dbf07cc71be14cf68d0d346c8dafb5d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:10:38.725723 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 27817
I20260925 10:10:38.730935 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 27892
I20260925 10:10:38.735522 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 27967
2026-09-25T10:10:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2953 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:10:38 dist-test-slave-s6tm krb5kdc[28106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:10:38 dist-test-slave-s6tm krb5kdc[28106](info): set up 2 sockets
Sep 25 10:10:38 dist-test-slave-s6tm krb5kdc[28106](info): commencing operation
krb5kdc: starting...
W20260925 10:10:39.789527 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:10:39 dist-test-slave-s6tm krb5kdc[28106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:10:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:10:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:10:39.912833 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:46091
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:46091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:10:39.996213 28119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:39.996556 28119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:39.996606 28119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:39.998155 28119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:10:39.998214 28119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:39.998229 28119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:39.998242 28119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:10:39.998255 28119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:10:40.000232 28119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:46091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:46091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:40.000551 28119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:40.000835 28119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:40.006421 28124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:40.006384 28127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:40.006414 28125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:40.006752 28119 server_base.cc:1034] running on GCE node
I20260925 10:10:40.007009 28119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:40.007293 28119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:40.008545 28119 hybrid_clock.cc:648] HybridClock initialized: now 1790331040008525 us; error 39 us; skew 500 ppm
Sep 25 10:10:40 dist-test-slave-s6tm krb5kdc[28106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331040, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:40.011224 28119 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:10:40.011725 28119 webserver.cc:460] Webserver started at http://127.14.41.126:46799/ using document root <none> and password file <none>
I20260925 10:10:40.011983 28119 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:40.012044 28119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:40.012149 28119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:10:40.013232 28119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "41a9c89e7e714bbbaf193a4d5c4eb598"
format_stamp: "Formatted at 2026-09-25 10:10:40 on dist-test-slave-s6tm"
server_key: "d36b2e01e7106ea159043230f8fb269c"
server_key_iv: "0682152e49ed212c2c39423c1882df16"
server_key_version: "encryptionkey@0"
I20260925 10:10:40.013550 28119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "41a9c89e7e714bbbaf193a4d5c4eb598"
format_stamp: "Formatted at 2026-09-25 10:10:40 on dist-test-slave-s6tm"
server_key: "d36b2e01e7106ea159043230f8fb269c"
server_key_iv: "0682152e49ed212c2c39423c1882df16"
server_key_version: "encryptionkey@0"
I20260925 10:10:40.014974 28119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:10:40.016022 28134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:40.016206 28119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:10:40.016311 28119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "41a9c89e7e714bbbaf193a4d5c4eb598"
format_stamp: "Formatted at 2026-09-25 10:10:40 on dist-test-slave-s6tm"
server_key: "d36b2e01e7106ea159043230f8fb269c"
server_key_iv: "0682152e49ed212c2c39423c1882df16"
server_key_version: "encryptionkey@0"
I20260925 10:10:40.016501 28119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:40.023888 28119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:40.026249 28119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:40.026535 28119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:40.030572 28119 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:46091
I20260925 10:10:40.030607 28186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:46091 every 8 connection(s)
I20260925 10:10:40.030974 28119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:10:40.032037 28187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:10:40.034439 28187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598: Bootstrap starting.
I20260925 10:10:40.034998 28187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598: Neither blocks nor log segments found. Creating new log.
I20260925 10:10:40.035231 28187 log.cc:824] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598: Log is configured to *not* fsync() on all Append() calls
I20260925 10:10:40.036036 28187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598: No bootstrap required, opened a new log
I20260925 10:10:40.037807 28187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a9c89e7e714bbbaf193a4d5c4eb598" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46091 } }
I20260925 10:10:40.038009 28187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:10:40.038049 28187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a9c89e7e714bbbaf193a4d5c4eb598, State: Initialized, Role: FOLLOWER
I20260925 10:10:40.038115 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 28119
I20260925 10:10:40.038223 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:10:40.038255 28187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a9c89e7e714bbbaf193a4d5c4eb598" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46091 } }
I20260925 10:10:40.038352 28187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:10:40.038386 28187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:10:40.038444 14501 external_mini_cluster.cc:1428] Setting key f941042bcd3a448b732e181ad2d10cb6
I20260925 10:10:40.038470 28187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:10:40.039618 28187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a9c89e7e714bbbaf193a4d5c4eb598" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46091 } }
I20260925 10:10:40.039744 28187 leader_election.cc:302] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41a9c89e7e714bbbaf193a4d5c4eb598; no voters:
I20260925 10:10:40.039963 28187 leader_election.cc:288] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:10:40.040014 28192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:10:40.040164 28192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598 [term 1 LEADER]: Becoming Leader. State: Replica: 41a9c89e7e714bbbaf193a4d5c4eb598, State: Running, Role: LEADER
I20260925 10:10:40.040243 28187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:10:40.040338 28192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a9c89e7e714bbbaf193a4d5c4eb598" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46091 } }
I20260925 10:10:40.040776 28192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41a9c89e7e714bbbaf193a4d5c4eb598. Latest consensus state: current_term: 1 leader_uuid: "41a9c89e7e714bbbaf193a4d5c4eb598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a9c89e7e714bbbaf193a4d5c4eb598" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46091 } } }
I20260925 10:10:40.040783 28194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41a9c89e7e714bbbaf193a4d5c4eb598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a9c89e7e714bbbaf193a4d5c4eb598" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46091 } } }
I20260925 10:10:40.040939 28194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598 [sys.catalog]: This master's current role is: LEADER
I20260925 10:10:40.040932 28192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598 [sys.catalog]: This master's current role is: LEADER
I20260925 10:10:40.041218 28201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:10:40.041956 28201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:10:40.043803 28201 catalog_manager.cc:1347] Generated new cluster ID: 2e423bfef8d54c0cbe2c801d08033fa5
I20260925 10:10:40.043869 28201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:10:40 dist-test-slave-s6tm krb5kdc[28106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:10:40.049244 28201 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:10:40.049981 28201 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:10:40.050011 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:40.038954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43342 (local address 127.14.41.126:46091)
0925 10:10:40.039153 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:40.039163 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:40.039200 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:10:40.039298 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:40.039304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:40.039350 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:40.039465 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:40.040383 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:40.042038 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:40.043130 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:40.043784 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:40.043903 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:40.046372 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:40.046376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:40.046377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:40.046389 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:40.048187 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:40.048592 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:40.048600 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:40.048600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:40.048688 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:40.048922 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:40.048922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:40.048923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:40.049214 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:40.049380 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:40.049631 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 10:10:40.049783 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":114,"threads_started":1}
I20260925 10:10:40.051067 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:10:40.060885 28201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41a9c89e7e714bbbaf193a4d5c4eb598: Generated new TSK 0
I20260925 10:10:40.061136 28201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:10:40.105055 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 28119
2026-09-25T10:10:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1367 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:10:40 dist-test-slave-s6tm krb5kdc[28214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:10:40 dist-test-slave-s6tm krb5kdc[28214](info): set up 2 sockets
Sep 25 10:10:40 dist-test-slave-s6tm krb5kdc[28214](info): commencing operation
krb5kdc: starting...
W20260925 10:10:41.145582 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:10:41 dist-test-slave-s6tm krb5kdc[28214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:10:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:10:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:10:41.270815 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:41135
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:41135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:10:41.353421 28227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:41.353684 28227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:41.353724 28227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:41.354928 28227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:10:41.355019 28227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:41.355048 28227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:41.355082 28227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:10:41.355113 28227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:10:41.356665 28227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:41135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:41135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:41.356951 28227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:41.357194 28227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:41.362294 28232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:41.362289 28235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:41.362308 28233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:41.362361 28227 server_base.cc:1034] running on GCE node
I20260925 10:10:41.362778 28227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:41.363085 28227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:41.364260 28227 hybrid_clock.cc:648] HybridClock initialized: now 1790331041364247 us; error 42 us; skew 500 ppm
Sep 25 10:10:41 dist-test-slave-s6tm krb5kdc[28214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331041, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:41.366827 28227 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:10:41.367307 28227 webserver.cc:460] Webserver started at http://127.14.41.126:34679/ using document root <none> and password file <none>
I20260925 10:10:41.367561 28227 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:41.367628 28227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:41.367774 28227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:10:41.368932 28227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "6fbdec816fe74e14819774dffeac2f44"
format_stamp: "Formatted at 2026-09-25 10:10:41 on dist-test-slave-s6tm"
server_key: "9d6d6d2db1b14447df93e0e21871be6c"
server_key_iv: "d8b3283bb10a227a63e8b6b4018a8dbd"
server_key_version: "encryptionkey@0"
I20260925 10:10:41.369444 28227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "6fbdec816fe74e14819774dffeac2f44"
format_stamp: "Formatted at 2026-09-25 10:10:41 on dist-test-slave-s6tm"
server_key: "9d6d6d2db1b14447df93e0e21871be6c"
server_key_iv: "d8b3283bb10a227a63e8b6b4018a8dbd"
server_key_version: "encryptionkey@0"
I20260925 10:10:41.370946 28227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:10:41.371737 28242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:41.371968 28227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:10:41.372080 28227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "6fbdec816fe74e14819774dffeac2f44"
format_stamp: "Formatted at 2026-09-25 10:10:41 on dist-test-slave-s6tm"
server_key: "9d6d6d2db1b14447df93e0e21871be6c"
server_key_iv: "d8b3283bb10a227a63e8b6b4018a8dbd"
server_key_version: "encryptionkey@0"
I20260925 10:10:41.372164 28227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:41.383872 28227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:41.386269 28227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:41.386528 28227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:41.390759 28227 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:41135
I20260925 10:10:41.390794 28294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:41135 every 8 connection(s)
I20260925 10:10:41.391163 28227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:10:41.392235 28295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:10:41.394672 28295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44: Bootstrap starting.
I20260925 10:10:41.395278 28295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44: Neither blocks nor log segments found. Creating new log.
I20260925 10:10:41.395586 28295 log.cc:824] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44: Log is configured to *not* fsync() on all Append() calls
I20260925 10:10:41.395767 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 28227
I20260925 10:10:41.395967 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:10:41.396167 14501 external_mini_cluster.cc:1428] Setting key b74747079b9b6e6df5b9cac8325b9446
I20260925 10:10:41.396449 28295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44: No bootstrap required, opened a new log
I20260925 10:10:41.398612 28295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fbdec816fe74e14819774dffeac2f44" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 41135 } }
I20260925 10:10:41.398803 28295 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:10:41.398837 28295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fbdec816fe74e14819774dffeac2f44, State: Initialized, Role: FOLLOWER
I20260925 10:10:41.398988 28295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fbdec816fe74e14819774dffeac2f44" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 41135 } }
I20260925 10:10:41.399087 28295 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:10:41.399119 28295 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:10:41.399165 28295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:10:41.399895 28295 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fbdec816fe74e14819774dffeac2f44" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 41135 } }
I20260925 10:10:41.400027 28295 leader_election.cc:302] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fbdec816fe74e14819774dffeac2f44; no voters:
I20260925 10:10:41.400192 28295 leader_election.cc:288] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:10:41.400242 28300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:10:41.400446 28300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 1 LEADER]: Becoming Leader. State: Replica: 6fbdec816fe74e14819774dffeac2f44, State: Running, Role: LEADER
I20260925 10:10:41.400450 28295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:10:41.400662 28300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fbdec816fe74e14819774dffeac2f44" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 41135 } }
I20260925 10:10:41.402001 28302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fbdec816fe74e14819774dffeac2f44. Latest consensus state: current_term: 1 leader_uuid: "6fbdec816fe74e14819774dffeac2f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fbdec816fe74e14819774dffeac2f44" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 41135 } } }
I20260925 10:10:41.402005 28301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fbdec816fe74e14819774dffeac2f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fbdec816fe74e14819774dffeac2f44" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 41135 } } }
I20260925 10:10:41.402145 28301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [sys.catalog]: This master's current role is: LEADER
I20260925 10:10:41.402146 28302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [sys.catalog]: This master's current role is: LEADER
W20260925 10:10:41.402592 28315 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:10:41.402658 28315 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:10:41.402742 28316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:10:41.403177 28316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 10:10:41 dist-test-slave-s6tm krb5kdc[28214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:10:41.404791 28316 catalog_manager.cc:1347] Generated new cluster ID: 6b4420f847d8427191395d19ac8bfd59
I20260925 10:10:41.404846 28316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:10:41.408529 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:41.396706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58368 (local address 127.14.41.126:41135)
0925 10:10:41.396939 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:41.396949 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:41.396985 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:10:41.397795 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:41.397806 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:41.397873 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:41.398035 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:41.398651 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.400373 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.401554 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.402231 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.402343 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:41.405240 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:41.405244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:41.405244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:41.405256 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:41.406958 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:41.407326 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:41.407330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:41.407330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:41.407419 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:41.407627 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:41.407628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:41.407628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:41.407863 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:41.408058 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:41.408187 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 10:10:41.408328 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":112,"threads_started":1}
I20260925 10:10:41.420846 28316 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:10:41.421437 28316 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:10:41.426716 28316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44: Generated new TSK 0
I20260925 10:10:41.426896 28316 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
I20260925 10:10:41.459697 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:41135
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:10:41.579959 28321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:41.580250 28321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:41.580283 28321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:41.582008 28321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:41.582070 28321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:41.582111 28321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:10:41.584340 28321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:41135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:41.584631 28321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:41.584941 28321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:41.590523 28329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:41.590526 28327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:41.590538 28326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:41.590865 28321 server_base.cc:1034] running on GCE node
I20260925 10:10:41.591061 28321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:41.591281 28321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:41.592458 28321 hybrid_clock.cc:648] HybridClock initialized: now 1790331041592447 us; error 36 us; skew 500 ppm
Sep 25 10:10:41 dist-test-slave-s6tm krb5kdc[28214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331041, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:41.595122 28321 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:10:41.595618 28321 webserver.cc:460] Webserver started at http://127.14.41.65:41581/ using document root <none> and password file <none>
I20260925 10:10:41.595849 28321 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:41.595914 28321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:41.596094 28321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:10:41.597152 28321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "a47799273a7f445e9657d46fe248d153"
format_stamp: "Formatted at 2026-09-25 10:10:41 on dist-test-slave-s6tm"
server_key: "52bddc060f6dbb81ad291cecbf974072"
server_key_iv: "895a1f85d266a3a418944c0759a09686"
server_key_version: "encryptionkey@0"
I20260925 10:10:41.597512 28321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "a47799273a7f445e9657d46fe248d153"
format_stamp: "Formatted at 2026-09-25 10:10:41 on dist-test-slave-s6tm"
server_key: "52bddc060f6dbb81ad291cecbf974072"
server_key_iv: "895a1f85d266a3a418944c0759a09686"
server_key_version: "encryptionkey@0"
I20260925 10:10:41.598862 28321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:10:41.599550 28336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:41.599730 28321 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:10:41.599826 28321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "a47799273a7f445e9657d46fe248d153"
format_stamp: "Formatted at 2026-09-25 10:10:41 on dist-test-slave-s6tm"
server_key: "52bddc060f6dbb81ad291cecbf974072"
server_key_iv: "895a1f85d266a3a418944c0759a09686"
server_key_version: "encryptionkey@0"
I20260925 10:10:41.599906 28321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:41.615382 28321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:41.617811 28321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:41.617985 28321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:41.618238 28321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:10:41.618628 28321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:10:41.618672 28321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:41.618705 28321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:10:41.618726 28321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:41.624778 28321 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:36629
I20260925 10:10:41.624850 28449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:36629 every 8 connection(s)
I20260925 10:10:41.625454 28321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:10:41.625916 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 28321
I20260925 10:10:41.626031 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:10:41.626255 14501 external_mini_cluster.cc:1428] Setting key 7897f62c254791ab870336c695bd6a58
Sep 25 10:10:41 dist-test-slave-s6tm krb5kdc[28214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331041, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.41.66@KRBTEST.COM; defaulting to no policy
I20260925 10:10:41.636994 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:41.626421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:60979 (local address 127.14.41.126:41135)
0925 10:10:41.626476 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:41.626478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:41.626756 (+ 278us) server_negotiation.cc:407] Connection header received
0925 10:10:41.627549 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:41.627551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:41.627562 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:41.627619 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:41.629030 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.629846 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.630864 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.631235 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.631315 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:41.633930 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:41.633931 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:41.633932 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:41.633942 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:41.635564 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:41.636023 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:41.636025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:41.636025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:41.636077 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:41.636311 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:41.636312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:41.636312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:41.636419 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:41.636470 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:41.636847 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 10:10:41.636923 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:10:41.637214 28452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:41.626405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.65:60979)
0925 10:10:41.626597 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:41.626653 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:41.627315 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:41.627713 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:41.627718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:41.628614 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:41.628934 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:41.628940 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.629961 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.629962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:41.630772 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:41.630774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.631324 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.631325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:41.631432 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:41.632139 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:41.632148 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:41.633833 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:41.635674 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:41.635682 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:41.635685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:41.635916 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:41.636138 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:41.636139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:41.636139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:41.636242 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:41.636498 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:41.636505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:41.636699 (+ 194us) client_negotiation.cc:769] Sending connection context
0925 10:10:41.636816 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 10:10:41.636966 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260925 10:10:41.637981 28450 heartbeater.cc:347] Connected to a master server at 127.14.41.126:41135
I20260925 10:10:41.638091 28450 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.66@KRBTEST.COM" created.
I20260925 10:10:41.638347 28450 heartbeater.cc:511] Master 127.14.41.126:41135 requested a full tablet report, sending...
I20260925 10:10:41.639003 28259 ts_manager.cc:194] Registered new tserver with Master: a47799273a7f445e9657d46fe248d153 (127.14.41.65:36629)
I20260925 10:10:41.639978 28259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:60979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
I20260925 10:10:41.655151 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:41135
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:10:41.747462 28455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:41.747707 28455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:41.747749 28455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:41.748993 28455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:41.749066 28455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:41.749114 28455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:10:41.750604 28455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:41135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:41.750882 28455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:41.751127 28455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:41.756306 28460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:41.756312 28461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:41.756316 28463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:41.756503 28455 server_base.cc:1034] running on GCE node
I20260925 10:10:41.756708 28455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:41.756916 28455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:41.758056 28455 hybrid_clock.cc:648] HybridClock initialized: now 1790331041758044 us; error 29 us; skew 500 ppm
Sep 25 10:10:41 dist-test-slave-s6tm krb5kdc[28214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331041, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:41.760645 28455 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:10:41.761039 28455 webserver.cc:460] Webserver started at http://127.14.41.66:39707/ using document root <none> and password file <none>
I20260925 10:10:41.761240 28455 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:41.761282 28455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:41.761370 28455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:10:41.762391 28455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "aee59a1978db4fa6aebf715bf7a8c9a7"
format_stamp: "Formatted at 2026-09-25 10:10:41 on dist-test-slave-s6tm"
server_key: "c584063975a1e931a8912de11b13bbdf"
server_key_iv: "d083101db71ca3d1ac6c05d514caf7cf"
server_key_version: "encryptionkey@0"
I20260925 10:10:41.762725 28455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "aee59a1978db4fa6aebf715bf7a8c9a7"
format_stamp: "Formatted at 2026-09-25 10:10:41 on dist-test-slave-s6tm"
server_key: "c584063975a1e931a8912de11b13bbdf"
server_key_iv: "d083101db71ca3d1ac6c05d514caf7cf"
server_key_version: "encryptionkey@0"
I20260925 10:10:41.764062 28455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:10:41.764766 28470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:41.764945 28455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 10:10:41.765029 28455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "aee59a1978db4fa6aebf715bf7a8c9a7"
format_stamp: "Formatted at 2026-09-25 10:10:41 on dist-test-slave-s6tm"
server_key: "c584063975a1e931a8912de11b13bbdf"
server_key_iv: "d083101db71ca3d1ac6c05d514caf7cf"
server_key_version: "encryptionkey@0"
I20260925 10:10:41.765093 28455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:41.774088 28455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:41.776341 28455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:41.776458 28455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:41.776643 28455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:10:41.776948 28455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:10:41.776981 28455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:41.777033 28455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:10:41.777067 28455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:41.782924 28455 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:39791
I20260925 10:10:41.783020 28583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:39791 every 8 connection(s)
I20260925 10:10:41.783321 28455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:10:41 dist-test-slave-s6tm krb5kdc[28214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331041, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:10:41.790283 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 28455
I20260925 10:10:41.790413 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:10:41.790683 14501 external_mini_cluster.cc:1428] Setting key efae2c135f8bc31b82bb07cb313991f5
I20260925 10:10:41.793260 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:41.784078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:34895 (local address 127.14.41.126:41135)
0925 10:10:41.784142 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:41.784144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:41.784374 (+ 230us) server_negotiation.cc:407] Connection header received
0925 10:10:41.784951 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:41.784952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:41.784966 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:41.785053 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:41.786123 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.786638 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.787680 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.787874 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.787972 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:41.790151 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:41.790153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:41.790153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:41.790166 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:41.791758 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:41.792224 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:41.792225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:41.792226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:41.792294 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:41.792563 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:41.792564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:41.792564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:41.792717 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:41.792789 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:41.793034 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 10:10:41.793125 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 10:10:41.793777 28586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:41.784084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.66:34895)
0925 10:10:41.784249 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:41.784288 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:41.784835 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:41.785132 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:41.785137 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:41.785804 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:41.786015 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:41.786020 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.786751 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.786752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:41.787584 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:41.787586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.788014 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.788015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:41.788146 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:41.788684 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:41.788692 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:41.790038 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:41.791838 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:41.791845 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:41.791847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:41.792153 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:41.792373 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:41.792374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:41.792374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:41.792478 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:41.792782 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:41.792786 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:41.792909 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 10:10:41.792993 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 10:10:41.793199 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260925 10:10:41.794329 28584 heartbeater.cc:347] Connected to a master server at 127.14.41.126:41135
I20260925 10:10:41.794449 28584 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:41.794658 28584 heartbeater.cc:511] Master 127.14.41.126:41135 requested a full tablet report, sending...
I20260925 10:10:41.795078 28259 ts_manager.cc:194] Registered new tserver with Master: aee59a1978db4fa6aebf715bf7a8c9a7 (127.14.41.66:39791)
I20260925 10:10:41.795509 28259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:34895
WARNING: no policy specified for kudu/127.14.41.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
I20260925 10:10:41.819434 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:41135
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:10:41.903333 28589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:41.903548 28589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:41.903571 28589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:41.904814 28589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:41.904862 28589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:41.904913 28589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:10:41.906380 28589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:41135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:41.906636 28589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:41.906869 28589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:41.912073 28595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:41.912082 28594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:41.912102 28589 server_base.cc:1034] running on GCE node
W20260925 10:10:41.912086 28597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:41.912508 28589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:41.912765 28589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:41.913910 28589 hybrid_clock.cc:648] HybridClock initialized: now 1790331041913891 us; error 34 us; skew 500 ppm
Sep 25 10:10:41 dist-test-slave-s6tm krb5kdc[28214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331041, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:41.916359 28589 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:10:41.916803 28589 webserver.cc:460] Webserver started at http://127.14.41.67:44749/ using document root <none> and password file <none>
I20260925 10:10:41.917099 28589 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:41.917179 28589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:41.917321 28589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:10:41.918391 28589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "cd8b60eb31174018a28f19837fad5007"
format_stamp: "Formatted at 2026-09-25 10:10:41 on dist-test-slave-s6tm"
server_key: "848fb5d410402778aa5fc14b12bbd70e"
server_key_iv: "cac6aa31399db0b79efe0a13af80caca"
server_key_version: "encryptionkey@0"
I20260925 10:10:41.918764 28589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "cd8b60eb31174018a28f19837fad5007"
format_stamp: "Formatted at 2026-09-25 10:10:41 on dist-test-slave-s6tm"
server_key: "848fb5d410402778aa5fc14b12bbd70e"
server_key_iv: "cac6aa31399db0b79efe0a13af80caca"
server_key_version: "encryptionkey@0"
I20260925 10:10:41.920156 28589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:10:41.920890 28604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:41.921077 28589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:10:41.921171 28589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "cd8b60eb31174018a28f19837fad5007"
format_stamp: "Formatted at 2026-09-25 10:10:41 on dist-test-slave-s6tm"
server_key: "848fb5d410402778aa5fc14b12bbd70e"
server_key_iv: "cac6aa31399db0b79efe0a13af80caca"
server_key_version: "encryptionkey@0"
I20260925 10:10:41.921252 28589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:41.953037 28589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:41.955288 28589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:41.955458 28589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:41.955713 28589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:10:41.956099 28589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:10:41.956151 28589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:41.956195 28589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:10:41.956221 28589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:41.962298 28589 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:43651
I20260925 10:10:41.962342 28717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:43651 every 8 connection(s)
I20260925 10:10:41.962673 28589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:10:41.964735 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 28589
I20260925 10:10:41.964833 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:10:41.965011 14501 external_mini_cluster.cc:1428] Setting key aea59ffe3a6a0d528075eb613891fd24
Sep 25 10:10:41 dist-test-slave-s6tm krb5kdc[28214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331041, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:10:41.972808 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:41.963396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:49611 (local address 127.14.41.126:41135)
0925 10:10:41.963475 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:41.963477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:41.963669 (+ 192us) server_negotiation.cc:407] Connection header received
0925 10:10:41.964241 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:41.964242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:41.964252 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:41.964304 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:41.965386 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.966124 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.967137 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.967316 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.967368 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:41.969897 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:41.969899 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:41.969900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:41.969908 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:41.971449 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:41.971856 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:41.971857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:41.971857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:41.971909 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:41.972197 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:41.972197 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:41.972197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:41.972310 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:41.972376 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:41.972663 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 10:10:41.972738 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:10:41.973016 28720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:41.963359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.67:49611)
0925 10:10:41.963545 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:41.963584 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:41.964150 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:41.964361 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:41.964364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:41.965005 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:41.965280 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:41.965287 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.966212 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.966214 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:41.967059 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:41.967061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.967385 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.967386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:41.967494 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:41.968154 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:41.968163 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:41.969802 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:41.971540 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:41.971548 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:41.971550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:41.971790 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:41.972025 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:41.972026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:41.972027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:41.972140 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:41.972377 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:41.972382 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:41.972533 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 10:10:41.972626 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 10:10:41.972761 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":99,"threads_started":1}
I20260925 10:10:41.973590 28718 heartbeater.cc:347] Connected to a master server at 127.14.41.126:41135
I20260925 10:10:41.973692 28718 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:41.973855 28718 heartbeater.cc:511] Master 127.14.41.126:41135 requested a full tablet report, sending...
I20260925 10:10:41.974294 28259 ts_manager.cc:194] Registered new tserver with Master: cd8b60eb31174018a28f19837fad5007 (127.14.41.67:43651)
I20260925 10:10:41.974779 28259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:49611
I20260925 10:10:41.976833 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:10:41.984151 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:41.978216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58370 (local address 127.14.41.126:41135)
0925 10:10:41.978287 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:41.978290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:41.978373 (+ 83us) server_negotiation.cc:407] Connection header received
0925 10:10:41.978408 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:41.978408 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:41.978417 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:41.978464 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:41.979000 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.979484 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.980299 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.980481 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.980537 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:41.981832 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:41.981833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:41.981833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:41.981840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:41.983087 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:41.983400 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:41.983400 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:41.983401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:41.983449 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:41.983610 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:41.983611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:41.983611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:41.983750 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:41.983827 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:41.983982 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 10:10:41.984066 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:10:41.985947 28259 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:10:41.987576 28259 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:10:41.994549 28730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:41.990359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:39791 (local address 127.0.0.1:60680)
0925 10:10:41.990540 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:41.990584 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:41.990652 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:41.990969 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:41.990972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:41.990975 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:10:41.991062 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:41.991065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.992202 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.992203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:41.993274 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:41.993277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.994194 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.994195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:41.994290 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:41.994305 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 10:10:41.994377 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 10:10:41.994434 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260925 10:10:41.994717 28733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:41.990703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53678 (local address 127.14.41.67:43651)
0925 10:10:41.990878 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:41.990883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:41.990952 (+ 69us) server_negotiation.cc:407] Connection header received
0925 10:10:41.991084 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:41.991086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:41.991102 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:41.991154 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:10:41.991381 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.992672 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.993761 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.994308 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.994385 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:41.994440 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:41.994561 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 10:10:41.994615 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
I20260925 10:10:41.994778 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:41.990475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.14.41.66:39791)
0925 10:10:41.990696 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:41.990701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:41.990714 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:10:41.990759 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:41.990761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:41.990779 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:41.990868 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:10:41.991149 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.992102 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.993352 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.994101 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.994185 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:41.994250 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:41.994627 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 10:10:41.994681 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":112,"threads_started":1}
I20260925 10:10:41.994871 28729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:41.990362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:43651 (local address 127.0.0.1:53678)
0925 10:10:41.990897 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:41.990907 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:41.990963 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:41.991263 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:41.991264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:41.991266 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:10:41.991315 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:41.991317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.992788 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.992789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:41.993679 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:41.993682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.994401 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.994402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:41.994466 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:41.994470 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:10:41.994529 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 10:10:41.994571 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":57,"threads_started":1}
I20260925 10:10:41.995767 28652 tablet_service.cc:1467] Processing CreateTablet for tablet f5794c47a3a44a41bf062d1f15b3157c (DEFAULT_TABLE table=test-table [id=2371cb5698d844f8b0c41ea2be158cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:10:41.995838 28518 tablet_service.cc:1467] Processing CreateTablet for tablet f5794c47a3a44a41bf062d1f15b3157c (DEFAULT_TABLE table=test-table [id=2371cb5698d844f8b0c41ea2be158cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:10:41.996196 28652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5794c47a3a44a41bf062d1f15b3157c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:10:41.996219 28518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5794c47a3a44a41bf062d1f15b3157c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:10:41.999023 28736 tablet_bootstrap.cc:492] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7: Bootstrap starting.
I20260925 10:10:41.999174 28735 tablet_bootstrap.cc:492] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007: Bootstrap starting.
I20260925 10:10:41.999351 28732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:41.990655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:36629 (local address 127.0.0.1:57934)
0925 10:10:41.991175 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:41.991184 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:41.991230 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:41.991430 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:41.991431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:41.991433 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:10:41.991468 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:41.991469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.993047 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.993049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:41.994177 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:41.994180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.999005 (+ 4825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.999007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:41.999091 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:41.999098 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:10:41.999214 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 10:10:41.999270 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":73,"threads_started":1}
I20260925 10:10:41.999408 28734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:41.990884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57934 (local address 127.14.41.65:36629)
0925 10:10:41.991054 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:41.991058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:41.991290 (+ 232us) server_negotiation.cc:407] Connection header received
0925 10:10:41.991339 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:41.991341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:41.991357 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:41.991437 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:10:41.991548 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.992880 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.994269 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:41.995207 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:41.995297 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:41.995352 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:41.999264 (+ 3912us) server_negotiation.cc:299] Negotiation successful
0925 10:10:41.999326 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260925 10:10:41.999836 28735 tablet_bootstrap.cc:654] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007: Neither blocks nor log segments found. Creating new log.
I20260925 10:10:42.000346 28736 tablet_bootstrap.cc:654] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7: Neither blocks nor log segments found. Creating new log.
I20260925 10:10:42.000373 28735 log.cc:824] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007: Log is configured to *not* fsync() on all Append() calls
I20260925 10:10:42.000334 28384 tablet_service.cc:1467] Processing CreateTablet for tablet f5794c47a3a44a41bf062d1f15b3157c (DEFAULT_TABLE table=test-table [id=2371cb5698d844f8b0c41ea2be158cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:10:42.000753 28384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5794c47a3a44a41bf062d1f15b3157c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:10:42.000977 28736 log.cc:824] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7: Log is configured to *not* fsync() on all Append() calls
I20260925 10:10:42.001204 28735 tablet_bootstrap.cc:492] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007: No bootstrap required, opened a new log
I20260925 10:10:42.001291 28735 ts_tablet_manager.cc:1397] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:10:42.002102 28736 tablet_bootstrap.cc:492] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7: No bootstrap required, opened a new log
I20260925 10:10:42.002267 28736 ts_tablet_manager.cc:1397] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 10:10:42.003157 28735 raft_consensus.cc:348] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:42.003319 28735 raft_consensus.cc:374] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:10:42.003367 28735 raft_consensus.cc:729] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd8b60eb31174018a28f19837fad5007, State: Initialized, Role: FOLLOWER
I20260925 10:10:42.003552 28735 consensus_queue.cc:260] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:42.003849 28735 ts_tablet_manager.cc:1428] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:10:42.004009 28718 heartbeater.cc:503] Master 127.14.41.126:41135 was elected leader, sending a full tablet report...
I20260925 10:10:42.004276 28740 tablet_bootstrap.cc:492] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153: Bootstrap starting.
I20260925 10:10:42.004666 28736 raft_consensus.cc:348] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:42.004870 28736 raft_consensus.cc:374] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:10:42.004918 28736 raft_consensus.cc:729] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee59a1978db4fa6aebf715bf7a8c9a7, State: Initialized, Role: FOLLOWER
I20260925 10:10:42.004936 28740 tablet_bootstrap.cc:654] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153: Neither blocks nor log segments found. Creating new log.
I20260925 10:10:42.005034 28736 consensus_queue.cc:260] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:42.005313 28736 ts_tablet_manager.cc:1428] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 10:10:42.005462 28584 heartbeater.cc:503] Master 127.14.41.126:41135 was elected leader, sending a full tablet report...
I20260925 10:10:42.005501 28740 log.cc:824] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153: Log is configured to *not* fsync() on all Append() calls
I20260925 10:10:42.006325 28740 tablet_bootstrap.cc:492] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153: No bootstrap required, opened a new log
I20260925 10:10:42.006397 28740 ts_tablet_manager.cc:1397] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:10:42.007784 28740 raft_consensus.cc:348] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:42.007972 28740 raft_consensus.cc:374] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:10:42.008013 28740 raft_consensus.cc:729] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47799273a7f445e9657d46fe248d153, State: Initialized, Role: FOLLOWER
I20260925 10:10:42.008111 28740 consensus_queue.cc:260] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:42.008337 28740 ts_tablet_manager.cc:1428] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:10:42.008373 28450 heartbeater.cc:503] Master 127.14.41.126:41135 was elected leader, sending a full tablet report...
W20260925 10:10:42.033969 28585 tablet.cc:2369] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:10:42.072613 28741 raft_consensus.cc:482] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:10:42.072763 28741 raft_consensus.cc:504] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:42.073370 28741 leader_election.cc:288] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd8b60eb31174018a28f19837fad5007 (127.14.41.67:43651), a47799273a7f445e9657d46fe248d153 (127.14.41.65:36629)
I20260925 10:10:42.076335 28586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.073589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:36629 (local address 127.14.41.66:42813)
0925 10:10:42.073652 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:42.073665 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:42.073713 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:42.073865 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:42.073867 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:42.073868 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:10:42.073941 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:42.073943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.074631 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.074632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:42.075434 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:42.075437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.076065 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.076066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:42.076150 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.076153 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:10:42.076219 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 10:10:42.076261 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 10:10:42.076335 28734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.073670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:42813 (local address 127.14.41.65:36629)
0925 10:10:42.073720 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:42.073723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:42.073738 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:10:42.073771 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:42.073772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:42.073782 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:42.073861 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:10:42.074035 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.074545 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.075526 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.075985 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.076066 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.076076 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:42.076225 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 10:10:42.076261 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:10:42.076632 28733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.073688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:54479 (local address 127.14.41.67:43651)
0925 10:10:42.073741 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:42.073743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:42.073895 (+ 152us) server_negotiation.cc:407] Connection header received
0925 10:10:42.073970 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:42.073971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:42.073985 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:42.074045 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:10:42.074190 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.074707 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.075782 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.076231 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.076311 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.076319 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:42.076515 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 10:10:42.076568 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 10:10:42.076751 28404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5794c47a3a44a41bf062d1f15b3157c" candidate_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47799273a7f445e9657d46fe248d153" is_pre_election: true
I20260925 10:10:42.076853 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.073616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:43651 (local address 127.14.41.66:54479)
0925 10:10:42.073819 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:42.073827 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:42.073903 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:42.074048 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:42.074049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:42.074051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:10:42.074124 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:42.074125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.074784 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.074785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:42.075699 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:42.075701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.076321 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.076322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:42.076416 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.076421 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:10:42.076723 (+ 302us) client_negotiation.cc:240] Negotiation successful
0925 10:10:42.076772 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":129,"threads_started":1}
I20260925 10:10:42.076938 28404 raft_consensus.cc:2454] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aee59a1978db4fa6aebf715bf7a8c9a7 in term 0.
I20260925 10:10:42.077162 28472 leader_election.cc:302] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47799273a7f445e9657d46fe248d153, aee59a1978db4fa6aebf715bf7a8c9a7; no voters:
I20260925 10:10:42.077247 28672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5794c47a3a44a41bf062d1f15b3157c" candidate_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8b60eb31174018a28f19837fad5007" is_pre_election: true
I20260925 10:10:42.077306 28741 raft_consensus.cc:2781] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:10:42.077353 28741 raft_consensus.cc:482] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:10:42.077375 28741 raft_consensus.cc:3037] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:10:42.077409 28672 raft_consensus.cc:2454] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aee59a1978db4fa6aebf715bf7a8c9a7 in term 0.
I20260925 10:10:42.078241 28741 raft_consensus.cc:504] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:42.078382 28741 leader_election.cc:288] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [CANDIDATE]: Term 1 election: Requested vote from peers cd8b60eb31174018a28f19837fad5007 (127.14.41.67:43651), a47799273a7f445e9657d46fe248d153 (127.14.41.65:36629)
I20260925 10:10:42.078528 28672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5794c47a3a44a41bf062d1f15b3157c" candidate_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8b60eb31174018a28f19837fad5007"
I20260925 10:10:42.078614 28404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5794c47a3a44a41bf062d1f15b3157c" candidate_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47799273a7f445e9657d46fe248d153"
I20260925 10:10:42.078627 28672 raft_consensus.cc:3037] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:10:42.078678 28404 raft_consensus.cc:3037] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:10:42.079558 28672 raft_consensus.cc:2454] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aee59a1978db4fa6aebf715bf7a8c9a7 in term 1.
I20260925 10:10:42.079558 28404 raft_consensus.cc:2454] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aee59a1978db4fa6aebf715bf7a8c9a7 in term 1.
I20260925 10:10:42.079753 28474 leader_election.cc:302] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aee59a1978db4fa6aebf715bf7a8c9a7, cd8b60eb31174018a28f19837fad5007; no voters:
I20260925 10:10:42.079882 28741 raft_consensus.cc:2781] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:10:42.080092 28741 raft_consensus.cc:686] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [term 1 LEADER]: Becoming Leader. State: Replica: aee59a1978db4fa6aebf715bf7a8c9a7, State: Running, Role: LEADER
I20260925 10:10:42.080204 28741 consensus_queue.cc:237] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:42.081038 28259 catalog_manager.cc:5495] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to aee59a1978db4fa6aebf715bf7a8c9a7 (127.14.41.66). New cstate: current_term: 1 leader_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } health_report { overall_health: UNKNOWN } } }
I20260925 10:10:42.086802 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.084436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.14.41.66:39791)
0925 10:10:42.084485 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:42.084486 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:42.084494 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:10:42.084577 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:42.084577 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:42.084592 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:42.084648 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:10:42.084758 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.085225 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.086033 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.086268 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.086322 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.086509 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:10:42.086607 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:10:42.086673 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:42.086709 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 10:10:42.086728 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:10:42.088562 28404 raft_consensus.cc:1261] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Refusing update from remote peer aee59a1978db4fa6aebf715bf7a8c9a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:10:42.088562 28672 raft_consensus.cc:1261] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007 [term 1 FOLLOWER]: Refusing update from remote peer aee59a1978db4fa6aebf715bf7a8c9a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:10:42.088927 28741 consensus_queue.cc:1035] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:10:42.089087 28745 consensus_queue.cc:1035] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:10:42.092128 28749 mvcc.cc:205] Tried to move back new op lower bound from 7333195948391886848 to 7333195948361162752. Current Snapshot: MvccSnapshot[applied={T|T < 7333195948391886848}]
I20260925 10:10:42.092753 28750 mvcc.cc:205] Tried to move back new op lower bound from 7333195948391886848 to 7333195948361162752. Current Snapshot: MvccSnapshot[applied={T|T < 7333195948391886848}]
I20260925 10:10:42.094167 28259 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:10:42.094265 28259 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:10:42.094969 28259 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44: Sending DeleteTablet for 3 replicas of tablet f5794c47a3a44a41bf062d1f15b3157c
I20260925 10:10:42.095350 28384 tablet_service.cc:1514] Processing DeleteTablet for tablet f5794c47a3a44a41bf062d1f15b3157c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:10:42 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:57934
I20260925 10:10:42.095331 28652 tablet_service.cc:1514] Processing DeleteTablet for tablet f5794c47a3a44a41bf062d1f15b3157c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:10:42 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:53678
I20260925 10:10:42.095456 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 28321
I20260925 10:10:42.095557 28518 tablet_service.cc:1514] Processing DeleteTablet for tablet f5794c47a3a44a41bf062d1f15b3157c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:10:42 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:60680
I20260925 10:10:42.096338 28759 tablet_replica.cc:330] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7: stopping tablet replica
I20260925 10:10:42.096463 28759 raft_consensus.cc:2229] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:10:42.096586 28759 raft_consensus.cc:2258] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:10:42.096733 28758 tablet_replica.cc:330] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007: stopping tablet replica
I20260925 10:10:42.096833 28758 raft_consensus.cc:2229] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:10:42.096944 28758 raft_consensus.cc:2258] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:10:42.097213 28759 ts_tablet_manager.cc:1905] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:10:42.098876 28759 ts_tablet_manager.cc:1918] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:10:42.099004 28759 log.cc:1196] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/f5794c47a3a44a41bf062d1f15b3157c
I20260925 10:10:42.099236 28758 ts_tablet_manager.cc:1905] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:10:42.099313 28759 ts_tablet_manager.cc:1939] T f5794c47a3a44a41bf062d1f15b3157c P aee59a1978db4fa6aebf715bf7a8c9a7: Deleting consensus metadata
I20260925 10:10:42.099725 28244 catalog_manager.cc:4841] TS aee59a1978db4fa6aebf715bf7a8c9a7 (127.14.41.66:39791): tablet f5794c47a3a44a41bf062d1f15b3157c (table test-table [id=2371cb5698d844f8b0c41ea2be158cb9]) successfully deleted
I20260925 10:10:42.101259 28758 ts_tablet_manager.cc:1918] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:10:42.101367 28758 log.cc:1196] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/f5794c47a3a44a41bf062d1f15b3157c
I20260925 10:10:42.101827 28758 ts_tablet_manager.cc:1939] T f5794c47a3a44a41bf062d1f15b3157c P cd8b60eb31174018a28f19837fad5007: Deleting consensus metadata
I20260925 10:10:42.102252 28246 catalog_manager.cc:4841] TS cd8b60eb31174018a28f19837fad5007 (127.14.41.67:43651): tablet f5794c47a3a44a41bf062d1f15b3157c (table test-table [id=2371cb5698d844f8b0c41ea2be158cb9]) successfully deleted
W20260925 10:10:42.103420 28244 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.41.65:36629 (error 108)
I20260925 10:10:42.103720 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 28455
I20260925 10:10:42.104353 28732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.103820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:36629 (local address 127.0.0.1:39220)
0925 10:10:42.104270 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:42.104314 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.65:36629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427}
W20260925 10:10:42.104681 28244 catalog_manager.cc:4584] TS a47799273a7f445e9657d46fe248d153 (127.14.41.65:36629): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f5794c47a3a44a41bf062d1f15b3157c: Network error: Client connection negotiation failed: client connection to 127.14.41.65:36629: connect: Connection refused (error 111)
I20260925 10:10:42.109052 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 28589
I20260925 10:10:42.113821 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 28227
I20260925 10:10:42.119140 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:41135
--webserver_interface=127.14.41.126
--webserver_port=34679
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:41135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:10:42.205209 28761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:42.205462 28761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:42.205488 28761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:42.206795 28761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:10:42.206856 28761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:42.206871 28761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:42.206887 28761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:10:42.206899 28761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:10:42.208674 28761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:41135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:41135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=34679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:42.208971 28761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:42.209244 28761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:42.220471 28768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:42.220466 28767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:42.220463 28770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:42.220511 28761 server_base.cc:1034] running on GCE node
I20260925 10:10:42.221040 28761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:42.221345 28761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:42.222499 28761 hybrid_clock.cc:648] HybridClock initialized: now 1790331042222479 us; error 28 us; skew 500 ppm
Sep 25 10:10:42 dist-test-slave-s6tm krb5kdc[28214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331042, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:42.231420 28761 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:10:42.232010 28761 webserver.cc:460] Webserver started at http://127.14.41.126:34679/ using document root <none> and password file <none>
I20260925 10:10:42.232245 28761 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:42.232301 28761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:42.233741 28761 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:10:42.234408 28777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:42.234606 28761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:10:42.234702 28761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "6fbdec816fe74e14819774dffeac2f44"
format_stamp: "Formatted at 2026-09-25 10:10:41 on dist-test-slave-s6tm"
server_key: "9d6d6d2db1b14447df93e0e21871be6c"
server_key_iv: "d8b3283bb10a227a63e8b6b4018a8dbd"
server_key_version: "encryptionkey@0"
I20260925 10:10:42.235118 28761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:42.268162 28761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:42.270926 28761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:42.271065 28761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:42.275566 28761 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:41135
I20260925 10:10:42.275602 28829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:41135 every 8 connection(s)
I20260925 10:10:42.276073 28761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:10:42.277858 28830 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:10:42.278558 28830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44: Bootstrap starting.
I20260925 10:10:42.284768 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 28761
I20260925 10:10:42.285180 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:36629
--local_ip_for_outbound_sockets=127.14.41.65
--tserver_master_addrs=127.14.41.126:41135
--webserver_port=41581
--webserver_interface=127.14.41.65
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:10:42.287273 28830 log.cc:824] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44: Log is configured to *not* fsync() on all Append() calls
I20260925 10:10:42.290282 28830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:10:42.290665 28830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44: Bootstrap complete.
I20260925 10:10:42.294902 28830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fbdec816fe74e14819774dffeac2f44" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 41135 } }
I20260925 10:10:42.295328 28830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fbdec816fe74e14819774dffeac2f44, State: Initialized, Role: FOLLOWER
I20260925 10:10:42.295529 28830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fbdec816fe74e14819774dffeac2f44" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 41135 } }
I20260925 10:10:42.295624 28830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:10:42.295691 28830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:10:42.295758 28830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:10:42.297609 28830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fbdec816fe74e14819774dffeac2f44" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 41135 } }
I20260925 10:10:42.297744 28830 leader_election.cc:302] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fbdec816fe74e14819774dffeac2f44; no voters:
I20260925 10:10:42.298045 28830 leader_election.cc:288] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:10:42.298092 28834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:10:42.298267 28834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 2 LEADER]: Becoming Leader. State: Replica: 6fbdec816fe74e14819774dffeac2f44, State: Running, Role: LEADER
I20260925 10:10:42.298372 28830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:10:42.298386 28834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fbdec816fe74e14819774dffeac2f44" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 41135 } }
I20260925 10:10:42.298715 28835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6fbdec816fe74e14819774dffeac2f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fbdec816fe74e14819774dffeac2f44" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 41135 } } }
I20260925 10:10:42.298796 28835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [sys.catalog]: This master's current role is: LEADER
I20260925 10:10:42.299052 28836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fbdec816fe74e14819774dffeac2f44. Latest consensus state: current_term: 2 leader_uuid: "6fbdec816fe74e14819774dffeac2f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fbdec816fe74e14819774dffeac2f44" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 41135 } } }
I20260925 10:10:42.299201 28836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [sys.catalog]: This master's current role is: LEADER
I20260925 10:10:42.299259 28840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:10:42.300203 28840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:10:42.300311 28840 catalog_manager.cc:1259] Loaded cluster ID: 6b4420f847d8427191395d19ac8bfd59
I20260925 10:10:42.300576 28840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:10:42.300874 28840 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 10:10:42.305940 28840 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 10:10:42.308781 28840 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:10:42.309018 28840 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44: Loaded TSK: 0
I20260925 10:10:42.309248 28840 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:10:42.379199 28832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:42.379439 28832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:42.379465 28832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:42.380937 28832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:42.380986 28832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:42.381016 28832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:10:42.382576 28832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:36629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=41581
--tserver_master_addrs=127.14.41.126:41135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:42.382848 28832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:42.383126 28832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:42.388387 28860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:42.388419 28857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:42.388480 28858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:42.388756 28832 server_base.cc:1034] running on GCE node
I20260925 10:10:42.388978 28832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:42.389271 28832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:42.390450 28832 hybrid_clock.cc:648] HybridClock initialized: now 1790331042390469 us; error 66 us; skew 500 ppm
Sep 25 10:10:42 dist-test-slave-s6tm krb5kdc[28214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331042, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:42.393606 28832 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:10:42.394142 28832 webserver.cc:460] Webserver started at http://127.14.41.65:41581/ using document root <none> and password file <none>
I20260925 10:10:42.394397 28832 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:42.394456 28832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:42.395987 28832 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:10:42.396739 28867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:42.396984 28832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:10:42.397081 28832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "a47799273a7f445e9657d46fe248d153"
format_stamp: "Formatted at 2026-09-25 10:10:41 on dist-test-slave-s6tm"
server_key: "52bddc060f6dbb81ad291cecbf974072"
server_key_iv: "895a1f85d266a3a418944c0759a09686"
server_key_version: "encryptionkey@0"
I20260925 10:10:42.397470 28832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:42.411617 28832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:42.413950 28832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:42.414115 28832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:42.414351 28832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:10:42.414810 28874 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 10:10:42.416759 28832 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 10:10:42.416821 28832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 10:10:42.416872 28832 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 10:10:42.417510 28832 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 10:10:42.417553 28832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 10:10:42.417631 28874 tablet_bootstrap.cc:492] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153: Bootstrap starting.
I20260925 10:10:42.424892 28832 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:36629
I20260925 10:10:42.425038 28981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:36629 every 8 connection(s)
I20260925 10:10:42.425333 28832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:10:42.429140 28874 log.cc:824] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153: Log is configured to *not* fsync() on all Append() calls
I20260925 10:10:42.431757 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 28832
I20260925 10:10:42.431969 28874 tablet_bootstrap.cc:492] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:10:42.432103 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:39791
--local_ip_for_outbound_sockets=127.14.41.66
--tserver_master_addrs=127.14.41.126:41135
--webserver_port=39707
--webserver_interface=127.14.41.66
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:10:42.432260 28874 tablet_bootstrap.cc:492] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153: Bootstrap complete.
I20260925 10:10:42.434039 28874 ts_tablet_manager.cc:1397] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.003s
Sep 25 10:10:42 dist-test-slave-s6tm krb5kdc[28214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331042, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:10:42.436010 28874 raft_consensus.cc:348] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:42.436642 28874 raft_consensus.cc:729] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47799273a7f445e9657d46fe248d153, State: Initialized, Role: FOLLOWER
I20260925 10:10:42.436815 28874 consensus_queue.cc:260] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:42.437098 28874 ts_tablet_manager.cc:1428] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:10:42.439993 28984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.426060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.65:50963)
0925 10:10:42.426249 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:42.426293 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:42.426802 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:42.428596 (+ 1794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:42.428600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:42.428907 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:42.429101 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:42.429107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.430462 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.430463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:42.431311 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:42.431314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.432224 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.432226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:42.432407 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.433124 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:42.433135 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:42.435058 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:42.437960 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:42.437967 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:42.437969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:42.438274 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:42.438579 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:42.438579 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:42.438580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:42.438713 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:42.439284 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:42.439290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:42.439454 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 10:10:42.439562 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:10:42.439744 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":111,"threads_started":1}
I20260925 10:10:42.440469 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.426147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:50963 (local address 127.14.41.126:41135)
0925 10:10:42.428141 (+ 1994us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:42.428149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:42.428183 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:10:42.428298 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:42.428304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:42.428348 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:42.428494 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:42.429234 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.430331 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.431411 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.432085 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.432726 (+ 641us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.435188 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:42.435194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:42.435194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:42.435208 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:42.437818 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:42.438368 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:42.438372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:42.438373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:42.438506 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:42.438784 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:42.438785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:42.438785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:42.439203 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:42.439287 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:42.439650 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 10:10:42.439819 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1931,"thread_start_us":91,"threads_started":1}
I20260925 10:10:42.441094 28982 heartbeater.cc:347] Connected to a master server at 127.14.41.126:41135
I20260925 10:10:42.441205 28982 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:42.441419 28982 heartbeater.cc:511] Master 127.14.41.126:41135 requested a full tablet report, sending...
I20260925 10:10:42.442310 28794 ts_manager.cc:194] Registered new tserver with Master: a47799273a7f445e9657d46fe248d153 (127.14.41.65:36629)
I20260925 10:10:42.443413 28794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:50963
I20260925 10:10:42.444597 28982 heartbeater.cc:503] Master 127.14.41.126:41135 was elected leader, sending a full tablet report...
Sep 25 10:10:42 dist-test-slave-s6tm krb5kdc[28214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331042, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for kudu/127.14.41.65@KRBTEST.COM
I20260925 10:10:42.452880 28989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.442872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:36629 (local address 127.0.0.1:39222)
0925 10:10:42.443103 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:42.443136 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:42.443195 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:42.444204 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:42.444211 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:42.444539 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:42.444604 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:42.444606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.446147 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.446148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:42.446883 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:42.446886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.447611 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.447612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:42.447701 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.448284 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:42.448288 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:42.449687 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:42.451710 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:42.451717 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:42.451719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:42.451919 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:42.452205 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:42.452206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:42.452206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:42.452269 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:42.452578 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:42.452584 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:42.452632 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 10:10:42.452701 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 10:10:42.452807 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":96,"threads_started":1}
I20260925 10:10:42.453079 28990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.442962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.14.41.65:36629)
0925 10:10:42.444039 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:42.444044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:42.444054 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:10:42.444088 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:42.444091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:42.444104 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:42.444172 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:42.444846 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.446038 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.446980 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.447504 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.447580 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.449773 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:42.449783 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:42.449783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:42.449791 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:42.451617 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:42.452069 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:42.452072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:42.452073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:42.452139 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:42.452341 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:42.452342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:42.452342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:42.452513 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:42.452595 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:42.452752 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 10:10:42.452840 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1047,"thread_start_us":89,"threads_started":1}
I20260925 10:10:42.453502 28916 tablet_service.cc:1514] Processing DeleteTablet for tablet f5794c47a3a44a41bf062d1f15b3157c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:10:42 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:39222
I20260925 10:10:42.454232 28991 tablet_replica.cc:330] stopping tablet replica
I20260925 10:10:42.454366 28991 raft_consensus.cc:2229] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:10:42.454442 28991 raft_consensus.cc:2258] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:10:42.454808 28991 ts_tablet_manager.cc:1905] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:10:42.457127 28991 ts_tablet_manager.cc:1918] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:10:42.457227 28991 log.cc:1196] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/wal/wals/f5794c47a3a44a41bf062d1f15b3157c
I20260925 10:10:42.457495 28991 ts_tablet_manager.cc:1939] T f5794c47a3a44a41bf062d1f15b3157c P a47799273a7f445e9657d46fe248d153: Deleting consensus metadata
I20260925 10:10:42.458109 28779 catalog_manager.cc:4841] TS a47799273a7f445e9657d46fe248d153 (127.14.41.65:36629): tablet f5794c47a3a44a41bf062d1f15b3157c (table test-table [id=2371cb5698d844f8b0c41ea2be158cb9]) successfully deleted
W20260925 10:10:42.527647 28987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:42.527905 28987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:42.527989 28987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:42.529420 28987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:42.529477 28987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:42.529510 28987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:10:42.531327 28987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:39791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=39707
--tserver_master_addrs=127.14.41.126:41135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:42.531633 28987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:42.531919 28987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:42.537899 28999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:42.537845 28996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:42.537845 28997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:42.538066 28987 server_base.cc:1034] running on GCE node
I20260925 10:10:42.538409 28987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:42.538738 28987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:42.539997 28987 hybrid_clock.cc:648] HybridClock initialized: now 1790331042539987 us; error 41 us; skew 500 ppm
Sep 25 10:10:42 dist-test-slave-s6tm krb5kdc[28214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331042, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:42.542915 28987 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:10:42.543594 28987 webserver.cc:460] Webserver started at http://127.14.41.66:39707/ using document root <none> and password file <none>
I20260925 10:10:42.543850 28987 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:42.543958 28987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:42.545768 28987 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:10:42.546532 29006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:42.546787 28987 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:10:42.546866 28987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "aee59a1978db4fa6aebf715bf7a8c9a7"
format_stamp: "Formatted at 2026-09-25 10:10:41 on dist-test-slave-s6tm"
server_key: "c584063975a1e931a8912de11b13bbdf"
server_key_iv: "d083101db71ca3d1ac6c05d514caf7cf"
server_key_version: "encryptionkey@0"
I20260925 10:10:42.547196 28987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:42.561295 28987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:42.563572 28987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:42.563699 28987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:42.563905 28987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:10:42.564304 28987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:10:42.564340 28987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:42.564371 28987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:10:42.564384 28987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:42.570659 28987 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:39791
I20260925 10:10:42.570694 29119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:39791 every 8 connection(s)
I20260925 10:10:42.571070 28987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:10:42 dist-test-slave-s6tm krb5kdc[28214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331042, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:10:42.579789 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 28987
I20260925 10:10:42.580158 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:43651
--local_ip_for_outbound_sockets=127.14.41.67
--tserver_master_addrs=127.14.41.126:41135
--webserver_port=44749
--webserver_interface=127.14.41.67
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:10:42.582652 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.572003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:58007 (local address 127.14.41.126:41135)
0925 10:10:42.572064 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:42.572071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:42.572223 (+ 152us) server_negotiation.cc:407] Connection header received
0925 10:10:42.572898 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:42.572900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:42.572917 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:42.572980 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:42.574169 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.574770 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.576066 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.576274 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.576421 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.579112 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:42.579114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:42.579114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:42.579122 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:42.580949 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:42.581510 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:42.581512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:42.581512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:42.581591 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:42.581896 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:42.581897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:42.581898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:42.582090 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:42.582166 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:42.582425 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 10:10:42.582544 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 10:10:42.582692 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.571820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.66:58007)
0925 10:10:42.572028 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:42.572099 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:42.572768 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:42.573070 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:42.573074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:42.573827 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:42.574061 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:42.574066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.574875 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.574876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:42.575878 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:42.575881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.576491 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.576492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:42.576612 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.577187 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:42.577197 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:42.578989 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:42.581050 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:42.581058 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:42.581060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:42.581404 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:42.581697 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:42.581698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:42.581698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:42.581818 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:42.582159 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:42.582164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:42.582292 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 10:10:42.582380 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 10:10:42.582517 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":70,"threads_started":1}
I20260925 10:10:42.583287 29120 heartbeater.cc:347] Connected to a master server at 127.14.41.126:41135
I20260925 10:10:42.583428 29120 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:42.583638 29120 heartbeater.cc:511] Master 127.14.41.126:41135 requested a full tablet report, sending...
I20260925 10:10:42.584074 28794 ts_manager.cc:194] Registered new tserver with Master: aee59a1978db4fa6aebf715bf7a8c9a7 (127.14.41.66:39791)
I20260925 10:10:42.584452 28794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:58007
W20260925 10:10:42.666294 29123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:42.666518 29123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:42.666543 29123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:42.667761 29123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:42.667811 29123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:42.667841 29123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:10:42.669435 29123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:43651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=44749
--tserver_master_addrs=127.14.41.126:41135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:42.669708 29123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:42.669943 29123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:42.675241 29129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:42.675241 29131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:42.675244 29128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:42.675589 29123 server_base.cc:1034] running on GCE node
I20260925 10:10:42.675836 29123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:42.676120 29123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:42.677295 29123 hybrid_clock.cc:648] HybridClock initialized: now 1790331042677244 us; error 46 us; skew 500 ppm
Sep 25 10:10:42 dist-test-slave-s6tm krb5kdc[28214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331042, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:42.679886 29123 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:10:42.680440 29123 webserver.cc:460] Webserver started at http://127.14.41.67:44749/ using document root <none> and password file <none>
I20260925 10:10:42.680702 29123 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:42.680761 29123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:42.682155 29123 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:10:42.682739 29138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:42.682896 29123 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:42.682991 29123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "cd8b60eb31174018a28f19837fad5007"
format_stamp: "Formatted at 2026-09-25 10:10:41 on dist-test-slave-s6tm"
server_key: "848fb5d410402778aa5fc14b12bbd70e"
server_key_iv: "cac6aa31399db0b79efe0a13af80caca"
server_key_version: "encryptionkey@0"
I20260925 10:10:42.683357 29123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:42.700628 29123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:42.702925 29123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:42.703050 29123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:42.703235 29123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:10:42.703575 29123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:10:42.703636 29123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:42.703680 29123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:10:42.703707 29123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:42.709807 29123 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:43651
I20260925 10:10:42.709923 29251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:43651 every 8 connection(s)
I20260925 10:10:42.710266 29123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:10:42.716449 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 29123
Sep 25 10:10:42 dist-test-slave-s6tm krb5kdc[28214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331042, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:10:42.720847 28985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.711120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:43925 (local address 127.14.41.126:41135)
0925 10:10:42.711171 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:42.711172 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:42.711436 (+ 264us) server_negotiation.cc:407] Connection header received
0925 10:10:42.712499 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:42.712500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:42.712512 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:42.712573 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:42.713290 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.713795 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.714827 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.715010 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.715066 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.717618 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:42.717620 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:42.717621 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:42.717630 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:42.719160 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:42.719599 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:42.719601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:42.719601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:42.719662 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:42.720176 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:42.720177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:42.720178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:42.720291 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:42.720367 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:42.720656 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 10:10:42.720754 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:10:42.721047 29254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.711125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.67:43925)
0925 10:10:42.711282 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:42.711321 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:42.712396 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:42.712632 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:42.712636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:42.712926 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:42.713187 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:42.713193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.713886 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.713888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:42.714745 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:42.714748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.715106 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.715107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:42.715231 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.715755 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:42.715764 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:42.717468 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:42.719264 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:42.719271 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:42.719274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:42.719500 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:42.719970 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:42.719971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:42.719971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:42.720096 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:42.720360 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:42.720367 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:42.720515 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 10:10:42.720621 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 10:10:42.720790 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260925 10:10:42.721546 29252 heartbeater.cc:347] Connected to a master server at 127.14.41.126:41135
I20260925 10:10:42.721666 29252 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:42.721957 29252 heartbeater.cc:511] Master 127.14.41.126:41135 requested a full tablet report, sending...
I20260925 10:10:42.722381 28794 ts_manager.cc:194] Registered new tserver with Master: cd8b60eb31174018a28f19837fad5007 (127.14.41.67:43651)
I20260925 10:10:42.722771 28794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:43925
I20260925 10:10:42.723672 29256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.716870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38226 (local address 127.14.41.126:41135)
0925 10:10:42.717005 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:42.717007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:42.717108 (+ 101us) server_negotiation.cc:407] Connection header received
0925 10:10:42.717176 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:42.717177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:42.717190 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:42.717256 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:42.717908 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.718436 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.719210 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.719422 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.719506 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.720655 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:42.720657 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:42.720657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:42.720664 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:42.722542 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:42.722902 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:42.722903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:42.722904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:42.722957 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:42.723204 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:42.723205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:42.723205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:42.723323 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:42.723450 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:42.723549 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 10:10:42.723611 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260925 10:10:42.724452 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:10:42.727337 29256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.724802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38238 (local address 127.14.41.126:41135)
0925 10:10:42.724861 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:42.724863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:42.724961 (+ 98us) server_negotiation.cc:407] Connection header received
0925 10:10:42.725067 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:42.725068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:42.725115 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:42.725192 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:10:42.725311 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.725767 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.726531 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.726754 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.726828 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.727012 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:10:42.727120 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:10:42.727211 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:42.727239 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 10:10:42.727260 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 10:10:42.727667 28794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:10:42.729339 28794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:10:42.731566 28916 tablet_service.cc:1467] Processing CreateTablet for tablet 42f21e0561474d83b04c118e54872835 (DEFAULT_TABLE table=test-table [id=a4e9b5c5b46942f2a87131ee34953af0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:10:42.731792 28916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42f21e0561474d83b04c118e54872835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:10:42.733196 28874 tablet_bootstrap.cc:492] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153: Bootstrap starting.
I20260925 10:10:42.733574 28874 tablet_bootstrap.cc:654] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153: Neither blocks nor log segments found. Creating new log.
I20260925 10:10:42.734588 28874 tablet_bootstrap.cc:492] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153: No bootstrap required, opened a new log
I20260925 10:10:42.734642 28874 ts_tablet_manager.cc:1397] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:10:42.734784 28874 raft_consensus.cc:348] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:42.734848 28874 raft_consensus.cc:374] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:10:42.734889 28874 raft_consensus.cc:729] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47799273a7f445e9657d46fe248d153, State: Initialized, Role: FOLLOWER
I20260925 10:10:42.734926 28874 consensus_queue.cc:260] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:42.735000 28874 ts_tablet_manager.cc:1428] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:42.735083 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.731198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:43651 (local address 127.0.0.1:36002)
0925 10:10:42.731375 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:42.731385 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:42.731439 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:42.731762 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:42.731763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:42.731766 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:10:42.731817 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:42.731818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.732948 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.732949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:42.733930 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:42.733933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.734787 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.734788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:42.734855 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.734874 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 10:10:42.734960 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 10:10:42.735006 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":127,"threads_started":1}
I20260925 10:10:42.735095 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.731391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36002 (local address 127.14.41.67:43651)
0925 10:10:42.731585 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:42.731590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:42.731601 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:10:42.731638 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:42.731641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:42.731658 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:42.731747 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:10:42.731894 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.732845 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.734019 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.734707 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.734775 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.734836 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:42.734978 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 10:10:42.735018 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":85,"threads_started":1}
I20260925 10:10:42.735886 29186 tablet_service.cc:1467] Processing CreateTablet for tablet 42f21e0561474d83b04c118e54872835 (DEFAULT_TABLE table=test-table [id=a4e9b5c5b46942f2a87131ee34953af0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:10:42.736261 29186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42f21e0561474d83b04c118e54872835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:10:42.737442 28989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.731142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:39791 (local address 127.0.0.1:57588)
0925 10:10:42.731243 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:42.731254 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:42.731288 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:42.733614 (+ 2326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:42.733615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:42.733617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:10:42.733663 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:42.733665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.735143 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.735144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:42.736225 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:42.736228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.737177 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.737178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:42.737255 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.737259 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:10:42.737342 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 10:10:42.737393 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 10:10:42.737522 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.731246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57588 (local address 127.14.41.66:39791)
0925 10:10:42.733427 (+ 2181us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:42.733431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:42.733443 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:10:42.733482 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:42.733484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:42.733502 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:42.733742 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:10:42.733822 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.734988 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.736335 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.737063 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.737136 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.737195 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:42.737363 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 10:10:42.737444 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2140,"thread_start_us":113,"threads_started":1}
I20260925 10:10:42.738155 29054 tablet_service.cc:1467] Processing CreateTablet for tablet 42f21e0561474d83b04c118e54872835 (DEFAULT_TABLE table=test-table [id=a4e9b5c5b46942f2a87131ee34953af0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:10:42.738495 29054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42f21e0561474d83b04c118e54872835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:10:42.739567 29262 tablet_bootstrap.cc:492] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007: Bootstrap starting.
I20260925 10:10:42.740310 29262 tablet_bootstrap.cc:654] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007: Neither blocks nor log segments found. Creating new log.
I20260925 10:10:42.740738 29262 log.cc:824] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007: Log is configured to *not* fsync() on all Append() calls
I20260925 10:10:42.741441 29262 tablet_bootstrap.cc:492] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007: No bootstrap required, opened a new log
I20260925 10:10:42.741513 29262 ts_tablet_manager.cc:1397] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 10:10:42.741532 29264 tablet_bootstrap.cc:492] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7: Bootstrap starting.
I20260925 10:10:42.742156 29264 tablet_bootstrap.cc:654] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7: Neither blocks nor log segments found. Creating new log.
I20260925 10:10:42.742756 29264 log.cc:824] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7: Log is configured to *not* fsync() on all Append() calls
I20260925 10:10:42.743542 29264 tablet_bootstrap.cc:492] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7: No bootstrap required, opened a new log
I20260925 10:10:42.743618 29264 ts_tablet_manager.cc:1397] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:10:42.743716 29262 raft_consensus.cc:348] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:42.743880 29262 raft_consensus.cc:374] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:10:42.743969 29262 raft_consensus.cc:729] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd8b60eb31174018a28f19837fad5007, State: Initialized, Role: FOLLOWER
I20260925 10:10:42.744135 29262 consensus_queue.cc:260] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:42.744442 29252 heartbeater.cc:503] Master 127.14.41.126:41135 was elected leader, sending a full tablet report...
I20260925 10:10:42.744427 29262 ts_tablet_manager.cc:1428] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 10:10:42.745675 29264 raft_consensus.cc:348] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:42.745803 29264 raft_consensus.cc:374] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:10:42.745824 29264 raft_consensus.cc:729] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee59a1978db4fa6aebf715bf7a8c9a7, State: Initialized, Role: FOLLOWER
I20260925 10:10:42.745898 29264 consensus_queue.cc:260] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:42.746148 29120 heartbeater.cc:503] Master 127.14.41.126:41135 was elected leader, sending a full tablet report...
I20260925 10:10:42.746148 29264 ts_tablet_manager.cc:1428] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 10:10:42.791463 28988 raft_consensus.cc:482] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:10:42.791641 28988 raft_consensus.cc:504] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:42.792021 28988 leader_election.cc:288] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aee59a1978db4fa6aebf715bf7a8c9a7 (127.14.41.66:39791), cd8b60eb31174018a28f19837fad5007 (127.14.41.67:43651)
I20260925 10:10:42.795343 28984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.792208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:39791 (local address 127.14.41.65:45227)
0925 10:10:42.792266 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:42.792280 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:42.792352 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:42.792722 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:42.792723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:42.792725 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:10:42.792818 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:42.792825 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.793570 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.793571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:42.794397 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:42.794400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.795052 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.795053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:42.795131 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.795165 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 10:10:42.795218 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 10:10:42.795249 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":52}
I20260925 10:10:42.795389 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.792516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:45227 (local address 127.14.41.66:39791)
0925 10:10:42.792576 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:42.792578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:42.792589 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:10:42.792622 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:42.792623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:42.792634 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:42.792706 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:10:42.792900 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.793454 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.794495 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.794977 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.795038 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.795045 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:42.795245 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 10:10:42.795296 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 10:10:42.795454 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.792540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:58659 (local address 127.14.41.67:43651)
0925 10:10:42.792588 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:42.792590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:42.792598 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:10:42.792688 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:42.792690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:42.792700 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:42.792760 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:10:42.793021 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.793470 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.794638 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.795056 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.795129 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.795137 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:42.795336 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 10:10:42.795380 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:10:42.795527 29268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.792283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:43651 (local address 127.14.41.65:58659)
0925 10:10:42.792521 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:42.792534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:42.792598 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:42.792783 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:42.792784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:42.792786 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:10:42.792916 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:42.792918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.793614 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.793615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:42.794526 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:42.794529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.795164 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.795166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:42.795252 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.795256 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:10:42.795316 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 10:10:42.795356 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":3,"thread_start_us":143,"threads_started":1}
I20260925 10:10:42.795718 29074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f21e0561474d83b04c118e54872835" candidate_uuid: "a47799273a7f445e9657d46fe248d153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" is_pre_election: true
I20260925 10:10:42.795890 29074 raft_consensus.cc:2454] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47799273a7f445e9657d46fe248d153 in term 0.
I20260925 10:10:42.795857 29206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f21e0561474d83b04c118e54872835" candidate_uuid: "a47799273a7f445e9657d46fe248d153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8b60eb31174018a28f19837fad5007" is_pre_election: true
I20260925 10:10:42.796072 29206 raft_consensus.cc:2454] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47799273a7f445e9657d46fe248d153 in term 0.
I20260925 10:10:42.796144 28869 leader_election.cc:302] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47799273a7f445e9657d46fe248d153, aee59a1978db4fa6aebf715bf7a8c9a7; no voters:
I20260925 10:10:42.796264 28988 raft_consensus.cc:2781] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:10:42.796346 28988 raft_consensus.cc:482] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:10:42.796378 28988 raft_consensus.cc:3037] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:10:42.797158 28988 raft_consensus.cc:504] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:42.797328 28988 leader_election.cc:288] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 1 election: Requested vote from peers aee59a1978db4fa6aebf715bf7a8c9a7 (127.14.41.66:39791), cd8b60eb31174018a28f19837fad5007 (127.14.41.67:43651)
I20260925 10:10:42.797515 29206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f21e0561474d83b04c118e54872835" candidate_uuid: "a47799273a7f445e9657d46fe248d153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8b60eb31174018a28f19837fad5007"
I20260925 10:10:42.797550 29074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f21e0561474d83b04c118e54872835" candidate_uuid: "a47799273a7f445e9657d46fe248d153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7"
I20260925 10:10:42.797598 29206 raft_consensus.cc:3037] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:10:42.797642 29074 raft_consensus.cc:3037] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:10:42.798396 29206 raft_consensus.cc:2454] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47799273a7f445e9657d46fe248d153 in term 1.
I20260925 10:10:42.798576 29074 raft_consensus.cc:2454] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47799273a7f445e9657d46fe248d153 in term 1.
I20260925 10:10:42.798585 28871 leader_election.cc:302] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47799273a7f445e9657d46fe248d153, cd8b60eb31174018a28f19837fad5007; no voters:
I20260925 10:10:42.798688 28988 raft_consensus.cc:2781] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:10:42.798900 28988 raft_consensus.cc:686] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 1 LEADER]: Becoming Leader. State: Replica: a47799273a7f445e9657d46fe248d153, State: Running, Role: LEADER
I20260925 10:10:42.799012 28988 consensus_queue.cc:237] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:42.800091 28794 catalog_manager.cc:5495] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 reported cstate change: term changed from 0 to 1, leader changed from <none> to a47799273a7f445e9657d46fe248d153 (127.14.41.65). New cstate: current_term: 1 leader_uuid: "a47799273a7f445e9657d46fe248d153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } health_report { overall_health: HEALTHY } } }
I20260925 10:10:42.812466 28990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.809738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.14.41.65:36629)
0925 10:10:42.809804 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:42.809806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:42.809816 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:10:42.809911 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:42.809911 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:42.809930 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:42.809997 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:10:42.810123 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.810663 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.811379 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:42.811618 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:42.811681 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:42.811874 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:10:42.812156 (+ 282us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:10:42.812237 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:42.812288 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 10:10:42.812329 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:10:42.814267 29206 raft_consensus.cc:1261] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007 [term 1 FOLLOWER]: Refusing update from remote peer a47799273a7f445e9657d46fe248d153: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:10:42.814271 29074 raft_consensus.cc:1261] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7 [term 1 FOLLOWER]: Refusing update from remote peer a47799273a7f445e9657d46fe248d153: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:10:42.814646 29269 consensus_queue.cc:1035] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:10:42.814822 28988 consensus_queue.cc:1035] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:10:42.817938 29274 mvcc.cc:205] Tried to move back new op lower bound from 7333195951364251648 to 7333195951305756672. Current Snapshot: MvccSnapshot[applied={T|T < 7333195951364251648}]
I20260925 10:10:42.818924 29273 mvcc.cc:205] Tried to move back new op lower bound from 7333195951364251648 to 7333195951305756672. Current Snapshot: MvccSnapshot[applied={T|T < 7333195951364251648 or (T in {7333195951364251648})}]
I20260925 10:10:42.820603 28793 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:10:42.820693 28793 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:10:42.821466 28793 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44: Sending DeleteTablet for 3 replicas of tablet 42f21e0561474d83b04c118e54872835
I20260925 10:10:42.821815 29054 tablet_service.cc:1514] Processing DeleteTablet for tablet 42f21e0561474d83b04c118e54872835 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:10:42 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:57588
I20260925 10:10:42.821867 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 28832
I20260925 10:10:42.821878 29186 tablet_service.cc:1514] Processing DeleteTablet for tablet 42f21e0561474d83b04c118e54872835 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:10:42 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:36002
I20260925 10:10:42.822582 29282 tablet_replica.cc:330] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7: stopping tablet replica
I20260925 10:10:42.822651 29283 tablet_replica.cc:330] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007: stopping tablet replica
I20260925 10:10:42.822687 29282 raft_consensus.cc:2229] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:10:42.822760 29283 raft_consensus.cc:2229] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:10:42.822794 29282 raft_consensus.cc:2258] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:10:42.822873 29283 raft_consensus.cc:2258] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:10:42.823319 29283 ts_tablet_manager.cc:1905] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:10:42.823853 29282 ts_tablet_manager.cc:1905] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:10:42.825244 29283 ts_tablet_manager.cc:1918] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:10:42.825357 29283 log.cc:1196] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/42f21e0561474d83b04c118e54872835
I20260925 10:10:42.825636 29283 ts_tablet_manager.cc:1939] T 42f21e0561474d83b04c118e54872835 P cd8b60eb31174018a28f19837fad5007: Deleting consensus metadata
I20260925 10:10:42.825874 29282 ts_tablet_manager.cc:1918] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:10:42.825976 29282 log.cc:1196] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/42f21e0561474d83b04c118e54872835
I20260925 10:10:42.826112 28781 catalog_manager.cc:4841] TS cd8b60eb31174018a28f19837fad5007 (127.14.41.67:43651): tablet 42f21e0561474d83b04c118e54872835 (table test-table [id=a4e9b5c5b46942f2a87131ee34953af0]) successfully deleted
I20260925 10:10:42.826272 29282 ts_tablet_manager.cc:1939] T 42f21e0561474d83b04c118e54872835 P aee59a1978db4fa6aebf715bf7a8c9a7: Deleting consensus metadata
I20260925 10:10:42.826885 28779 catalog_manager.cc:4841] TS aee59a1978db4fa6aebf715bf7a8c9a7 (127.14.41.66:39791): tablet 42f21e0561474d83b04c118e54872835 (table test-table [id=a4e9b5c5b46942f2a87131ee34953af0]) successfully deleted
W20260925 10:10:42.828842 28779 connection.cc:535] client connection to 127.14.41.65:36629 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 10:10:42.828958 28779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 10:10:42.829188 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 28987
I20260925 10:10:42.829473 28989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:42.829356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:36629 (local address 127.0.0.1:39240)
0925 10:10:42.829404 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:42.829435 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.65:36629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 10:10:42.829900 28779 catalog_manager.cc:4584] TS a47799273a7f445e9657d46fe248d153 (127.14.41.65:36629): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42f21e0561474d83b04c118e54872835: Network error: Client connection negotiation failed: client connection to 127.14.41.65:36629: connect: Connection refused (error 111)
I20260925 10:10:42.834838 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 29123
I20260925 10:10:42.839634 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 28761
I20260925 10:10:42.845484 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:41135
--webserver_interface=127.14.41.126
--webserver_port=34679
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:41135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:10:42.932096 29285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:42.932329 29285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:42.932355 29285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:42.933496 29285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:10:42.933540 29285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:42.933555 29285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:42.933568 29285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:10:42.933580 29285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:10:42.934993 29285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:41135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:41135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=34679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:42.935187 29285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:42.935364 29285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:42.940656 29290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:42.940801 29291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:42.941021 29285 server_base.cc:1034] running on GCE node
W20260925 10:10:42.940722 29293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:42.941392 29285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:42.941732 29285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:42.942907 29285 hybrid_clock.cc:648] HybridClock initialized: now 1790331042942882 us; error 37 us; skew 500 ppm
Sep 25 10:10:42 dist-test-slave-s6tm krb5kdc[28214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331042, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:42.945488 29285 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:10:42.945947 29285 webserver.cc:460] Webserver started at http://127.14.41.126:34679/ using document root <none> and password file <none>
I20260925 10:10:42.946187 29285 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:42.946259 29285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:42.947669 29285 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:10:42.948462 29300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:42.948637 29285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:10:42.948737 29285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "6fbdec816fe74e14819774dffeac2f44"
format_stamp: "Formatted at 2026-09-25 10:10:41 on dist-test-slave-s6tm"
server_key: "9d6d6d2db1b14447df93e0e21871be6c"
server_key_iv: "d8b3283bb10a227a63e8b6b4018a8dbd"
server_key_version: "encryptionkey@0"
I20260925 10:10:42.949147 29285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:42.967402 29285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:42.969416 29285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:42.969599 29285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:42.974058 29285 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:41135
I20260925 10:10:42.974097 29352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:41135 every 8 connection(s)
I20260925 10:10:42.974503 29285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:10:42.976238 29353 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:10:42.976977 29353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44: Bootstrap starting.
I20260925 10:10:42.980739 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 29285
I20260925 10:10:42.981159 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:36629
--local_ip_for_outbound_sockets=127.14.41.65
--tserver_master_addrs=127.14.41.126:41135
--webserver_port=41581
--webserver_interface=127.14.41.65
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:10:42.986284 29353 log.cc:824] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44: Log is configured to *not* fsync() on all Append() calls
I20260925 10:10:42.988842 29353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:10:42.989176 29353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44: Bootstrap complete.
I20260925 10:10:42.991446 29353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fbdec816fe74e14819774dffeac2f44" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 41135 } }
I20260925 10:10:42.991705 29353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fbdec816fe74e14819774dffeac2f44, State: Initialized, Role: FOLLOWER
I20260925 10:10:42.991894 29353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fbdec816fe74e14819774dffeac2f44" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 41135 } }
I20260925 10:10:42.992008 29353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:10:42.992092 29353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:10:42.992148 29353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 2 FOLLOWER]: Advancing to term 3
I20260925 10:10:42.993199 29353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fbdec816fe74e14819774dffeac2f44" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 41135 } }
I20260925 10:10:42.993330 29353 leader_election.cc:302] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fbdec816fe74e14819774dffeac2f44; no voters:
I20260925 10:10:42.993564 29353 leader_election.cc:288] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 10:10:42.993607 29357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 10:10:42.993733 29357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [term 3 LEADER]: Becoming Leader. State: Replica: 6fbdec816fe74e14819774dffeac2f44, State: Running, Role: LEADER
I20260925 10:10:42.993854 29353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:10:42.993860 29357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fbdec816fe74e14819774dffeac2f44" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 41135 } }
I20260925 10:10:42.994159 29357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fbdec816fe74e14819774dffeac2f44. Latest consensus state: current_term: 3 leader_uuid: "6fbdec816fe74e14819774dffeac2f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fbdec816fe74e14819774dffeac2f44" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 41135 } } }
I20260925 10:10:42.994230 29357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [sys.catalog]: This master's current role is: LEADER
I20260925 10:10:42.994370 29358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6fbdec816fe74e14819774dffeac2f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fbdec816fe74e14819774dffeac2f44" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 41135 } } }
I20260925 10:10:42.994447 29358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44 [sys.catalog]: This master's current role is: LEADER
I20260925 10:10:42.995733 29373 catalog_manager.cc:1259] Loaded cluster ID: 6b4420f847d8427191395d19ac8bfd59
I20260925 10:10:42.995783 29373 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44: loading cluster ID for follower catalog manager: success
W20260925 10:10:42.996037 29373 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 6fbdec816fe74e14819774dffeac2f44: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260925 10:10:42.996083 29373 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260925 10:10:42.996117 29362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:10:42.996495 29362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:10:42.996589 29362 catalog_manager.cc:1259] Loaded cluster ID: 6b4420f847d8427191395d19ac8bfd59
I20260925 10:10:42.996621 29362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 10:10:42.996770 29362 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f762ade09ad google::LogMessage::Fail() at ??:0
@ 0x7f762ade388a google::LogMessage::SendToLog() at ??:0
@ 0x7f762ade04de google::LogMessage::Flush() at ??:0
@ 0x7f762ade1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7632d010f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f7632d0a8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f762ccae0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f762cca3954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f762db076db start_thread at ??:0
@ 0x7f76298de61f clone at ??:0
W20260925 10:10:43.077584 29355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:43.077842 29355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:43.077869 29355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:43.079103 29355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:43.079147 29355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:43.079175 29355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:10:43.080761 29355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:36629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=41581
--tserver_master_addrs=127.14.41.126:41135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:43.081010 29355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:43.081255 29355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:43.086508 29390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:43.086464 29389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:43.086464 29392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:43.086822 29355 server_base.cc:1034] running on GCE node
I20260925 10:10:43.087047 29355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:43.087348 29355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:43.088529 29355 hybrid_clock.cc:648] HybridClock initialized: now 1790331043088512 us; error 39 us; skew 500 ppm
Sep 25 10:10:43 dist-test-slave-s6tm krb5kdc[28214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331043, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:43.091665 29355 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:10:43.092242 29355 webserver.cc:460] Webserver started at http://127.14.41.65:41581/ using document root <none> and password file <none>
I20260925 10:10:43.092520 29355 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:43.092628 29355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:43.094456 29355 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:10:43.095300 29399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:43.095525 29355 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:10:43.095631 29355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "a47799273a7f445e9657d46fe248d153"
format_stamp: "Formatted at 2026-09-25 10:10:41 on dist-test-slave-s6tm"
server_key: "52bddc060f6dbb81ad291cecbf974072"
server_key_iv: "895a1f85d266a3a418944c0759a09686"
server_key_version: "encryptionkey@0"
I20260925 10:10:43.096119 29355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:43.112732 29355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:43.114835 29355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:43.114948 29355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:43.115166 29355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:10:43.115674 29406 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 10:10:43.117257 29355 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 10:10:43.117319 29355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 10:10:43.117352 29355 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 10:10:43.118216 29355 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 10:10:43.118266 29355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 10:10:43.118352 29406 tablet_bootstrap.cc:492] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153: Bootstrap starting.
I20260925 10:10:43.125890 29355 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:36629
I20260925 10:10:43.125953 29513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:36629 every 8 connection(s)
I20260925 10:10:43.126346 29355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:10:43.127084 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 29355
I20260925 10:10:43.127439 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:39791
--local_ip_for_outbound_sockets=127.14.41.66
--tserver_master_addrs=127.14.41.126:41135
--webserver_port=39707
--webserver_interface=127.14.41.66
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:10:43.130187 29406 log.cc:824] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153: Log is configured to *not* fsync() on all Append() calls
I20260925 10:10:43.133155 29406 tablet_bootstrap.cc:492] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:10:43.133510 29406 tablet_bootstrap.cc:492] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153: Bootstrap complete.
I20260925 10:10:43.134872 29406 ts_tablet_manager.cc:1397] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.005s
I20260925 10:10:43.136343 29406 raft_consensus.cc:348] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:43.136754 29406 raft_consensus.cc:729] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47799273a7f445e9657d46fe248d153, State: Initialized, Role: FOLLOWER
I20260925 10:10:43.136914 29406 consensus_queue.cc:260] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:43.137219 29406 ts_tablet_manager.cc:1428] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260925 10:10:43.213011 29517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:43.213274 29517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:43.213320 29517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:43.214603 29517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:43.214673 29517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:43.214718 29517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:10:43.216311 29517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:39791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=39707
--tserver_master_addrs=127.14.41.126:41135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:43.216603 29517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:43.216851 29517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:43.222345 29525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:43.222319 29524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:43.222327 29527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:43.222630 29517 server_base.cc:1034] running on GCE node
I20260925 10:10:43.223053 29517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:43.223369 29517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:43.224546 29517 hybrid_clock.cc:648] HybridClock initialized: now 1790331043224532 us; error 34 us; skew 500 ppm
Sep 25 10:10:43 dist-test-slave-s6tm krb5kdc[28214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331043, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:43.227227 29517 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:10:43.227766 29517 webserver.cc:460] Webserver started at http://127.14.41.66:39707/ using document root <none> and password file <none>
I20260925 10:10:43.228101 29517 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:43.228156 29517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:43.229599 29517 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:10:43.230295 29534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:43.230494 29517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:10:43.230600 29517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "aee59a1978db4fa6aebf715bf7a8c9a7"
format_stamp: "Formatted at 2026-09-25 10:10:41 on dist-test-slave-s6tm"
server_key: "c584063975a1e931a8912de11b13bbdf"
server_key_iv: "d083101db71ca3d1ac6c05d514caf7cf"
server_key_version: "encryptionkey@0"
I20260925 10:10:43.230971 29517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260925 10:10:43.244166 29516 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:10:43.127114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.65:49709)
0925 10:10:43.127313 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:43.127366 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:43.128186 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:43.243910 (+115724us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":110,"threads_started":1}
W20260925 10:10:43.244452 29514 heartbeater.cc:644] Failed to heartbeat to 127.14.41.126:41135 (0 consecutive failures): Network error: Failed to ping master at 127.14.41.126:41135: Client connection negotiation failed: client connection to 127.14.41.126:41135: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 10:10:43.244809 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:43.244695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.65:44689)
0925 10:10:43.244745 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:43.244787 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 10:10:43.245134 29516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:43.245056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.65:50421)
0925 10:10:43.245097 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:43.245120 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 10:10:43.245267 29514 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:10:43.249867 29517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:43.251752 29517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:43.251915 29517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:43.252190 29517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:10:43.252518 29517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:10:43.252573 29517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:43.252614 29517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:10:43.252640 29517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:43.258875 29517 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:39791
I20260925 10:10:43.258916 29647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:39791 every 8 connection(s)
I20260925 10:10:43.259284 29517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:10:43.260344 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:43.259968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.66:54605)
0925 10:10:43.260117 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:43.260198 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
W20260925 10:10:43.260761 29648 heartbeater.cc:644] Failed to heartbeat to 127.14.41.126:41135 (0 consecutive failures): Network error: Failed to ping master at 127.14.41.126:41135: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
I20260925 10:10:43.261045 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:43.260947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.66:40361)
0925 10:10:43.261000 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:43.261027 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 10:10:43.261286 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:43.261234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.66:57131)
0925 10:10:43.261257 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:43.261275 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260925 10:10:43.261389 29648 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:10:43.263913 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 29517
I20260925 10:10:43.264470 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:43651
--local_ip_for_outbound_sockets=127.14.41.67
--tserver_master_addrs=127.14.41.126:41135
--webserver_port=44749
--webserver_interface=127.14.41.67
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:10:43.349313 29651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:43.349579 29651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:43.349608 29651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:43.350780 29651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:43.350821 29651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:43.350847 29651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:10:43.352356 29651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:43651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=44749
--tserver_master_addrs=127.14.41.126:41135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:43.352555 29651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:43.352736 29651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:43.357774 29657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:43.357779 29659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:43.358036 29656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:43.358105 29651 server_base.cc:1034] running on GCE node
I20260925 10:10:43.358372 29651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:43.358642 29651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:43.359812 29651 hybrid_clock.cc:648] HybridClock initialized: now 1790331043359807 us; error 36 us; skew 500 ppm
Sep 25 10:10:43 dist-test-slave-s6tm krb5kdc[28214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331043, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:43.362355 29651 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:10:43.362771 29651 webserver.cc:460] Webserver started at http://127.14.41.67:44749/ using document root <none> and password file <none>
I20260925 10:10:43.362977 29651 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:43.363021 29651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:43.364395 29651 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:10:43.365013 29666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:43.365206 29651 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:10:43.365288 29651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "cd8b60eb31174018a28f19837fad5007"
format_stamp: "Formatted at 2026-09-25 10:10:41 on dist-test-slave-s6tm"
server_key: "848fb5d410402778aa5fc14b12bbd70e"
server_key_iv: "cac6aa31399db0b79efe0a13af80caca"
server_key_version: "encryptionkey@0"
I20260925 10:10:43.365681 29651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:43.380810 29651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:43.382881 29651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:43.383041 29651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:43.383294 29651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:10:43.383621 29651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:10:43.383677 29651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:43.383721 29651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:10:43.383749 29651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:43.389896 29651 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:43651
I20260925 10:10:43.389969 29779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:43651 every 8 connection(s)
I20260925 10:10:43.390341 29651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:10:43.391458 29782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:43.391061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.67:34499)
0925 10:10:43.391223 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:43.391308 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
W20260925 10:10:43.391896 29780 heartbeater.cc:644] Failed to heartbeat to 127.14.41.126:41135 (0 consecutive failures): Network error: Failed to ping master at 127.14.41.126:41135: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
I20260925 10:10:43.392253 29782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:43.392145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.67:55657)
0925 10:10:43.392195 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:43.392222 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 10:10:43.392549 29782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:43.392458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.67:49643)
0925 10:10:43.392504 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:43.392527 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 10:10:43.392678 29780 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:10:43.400074 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 29651
I20260925 10:10:44.245919 29784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:44.245655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.65:49175)
0925 10:10:44.245811 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:44.245856 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":68,"threads_started":1}
I20260925 10:10:44.262027 29785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:44.261752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.66:48233)
0925 10:10:44.261931 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:44.261969 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":68,"threads_started":1}
I20260925 10:10:44.393304 29786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:44.393057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.67:38529)
0925 10:10:44.393198 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:44.393244 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260925 10:10:44.598824 29787 raft_consensus.cc:482] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:10:44.598946 29787 raft_consensus.cc:504] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:44.599277 29787 leader_election.cc:288] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aee59a1978db4fa6aebf715bf7a8c9a7 (127.14.41.66:39791), cd8b60eb31174018a28f19837fad5007 (127.14.41.67:43651)
Sep 25 10:10:44 dist-test-slave-s6tm krb5kdc[28214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331043, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.67@KRBTEST.COM
Sep 25 10:10:44 dist-test-slave-s6tm krb5kdc[28214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331043, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.66@KRBTEST.COM
I20260925 10:10:44.610040 29789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:44.599550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:54579 (local address 127.14.41.66:39791)
0925 10:10:44.599784 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:44.599799 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:44.599888 (+ 89us) server_negotiation.cc:407] Connection header received
0925 10:10:44.600447 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:44.600449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:44.600496 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:44.600671 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:44.601476 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:44.602356 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:44.603374 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:44.603772 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:44.603906 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:44.606556 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:44.606561 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:44.606562 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:44.606575 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:44.608294 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:44.608799 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:44.608802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:44.608802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:44.608897 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:44.609191 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:44.609192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:44.609192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:44.609404 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:44.609500 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:44.609775 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 10:10:44.609910 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":95,"threads_started":1}
I20260925 10:10:44.610047 29790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:44.599600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:35027 (local address 127.14.41.67:43651)
0925 10:10:44.599830 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:44.599842 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:44.599879 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:10:44.600446 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:44.600449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:44.600496 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:44.600667 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:44.601476 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:44.602344 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:44.603360 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:44.603766 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:44.603888 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:44.606238 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:44.606243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:44.606243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:44.606255 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:44.608270 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:44.608798 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:44.608801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:44.608802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:44.608897 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:44.609157 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:44.609158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:44.609159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:44.609404 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:44.609507 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:44.609778 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 10:10:44.609907 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":64,"threads_started":1}
I20260925 10:10:44.610015 29784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:44.599443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:43651 (local address 127.14.41.65:35027)
0925 10:10:44.599553 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:44.599567 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:44.599632 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:44.600715 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:44.600718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:44.601159 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:44.601355 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:44.601361 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:44.602464 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:44.602465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:44.603261 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:44.603263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:44.603856 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:44.603857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:44.604000 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:44.604611 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:44.604619 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:44.606113 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:44.608347 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:44.608354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:44.608357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:44.608710 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:44.608969 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:44.608970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:44.608970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:44.609088 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:44.609482 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:44.609488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:44.609611 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 10:10:44.609722 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 10:10:44.609920 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":39}
I20260925 10:10:44.610162 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:44.599443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:39791 (local address 127.14.41.65:54579)
0925 10:10:44.599638 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:44.599647 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:44.599688 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:44.600716 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:44.600719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:44.601158 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:44.601355 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:44.601361 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:44.602457 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:44.602459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:44.603263 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:44.603266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:44.603875 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:44.603877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:44.603999 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:44.604670 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:44.604674 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:44.606448 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:44.608408 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:44.608409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:44.608409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:44.608709 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:44.608976 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:44.608977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:44.608977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:44.609097 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:44.609482 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:44.609488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:44.609611 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 10:10:44.609721 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 10:10:44.610009 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":186,"thread_start_us":83,"threads_started":1}
W20260925 10:10:44.610921 29403 leader_election.cc:341] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd8b60eb31174018a28f19837fad5007 (127.14.41.67:43651): Not found: Tablet not found: 42f21e0561474d83b04c118e54872835
W20260925 10:10:44.611768 29401 leader_election.cc:341] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aee59a1978db4fa6aebf715bf7a8c9a7 (127.14.41.66:39791): Not found: Tablet not found: 42f21e0561474d83b04c118e54872835
I20260925 10:10:44.611837 29401 leader_election.cc:302] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a47799273a7f445e9657d46fe248d153; no voters: aee59a1978db4fa6aebf715bf7a8c9a7, cd8b60eb31174018a28f19837fad5007
I20260925 10:10:44.611989 29787 raft_consensus.cc:2726] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:10:45.246748 29791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:45.246482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.65:38333)
0925 10:10:45.246649 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:45.246698 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":98,"threads_started":1}
I20260925 10:10:45.262799 29792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:45.262578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.66:34771)
0925 10:10:45.262714 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:45.262755 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":58,"threads_started":1}
I20260925 10:10:45.394029 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:45.393787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.67:56235)
0925 10:10:45.393933 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:45.393977 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260925 10:10:46.247573 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:46.247321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.65:45311)
0925 10:10:46.247453 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:46.247525 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260925 10:10:46.263705 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:46.263456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.66:35607)
0925 10:10:46.263616 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:46.263653 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260925 10:10:46.394827 29796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:46.394589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.67:51575)
0925 10:10:46.394735 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:46.394780 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260925 10:10:46.572012 29797 raft_consensus.cc:482] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:10:46.572111 29797 raft_consensus.cc:504] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:46.572286 29797 leader_election.cc:288] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aee59a1978db4fa6aebf715bf7a8c9a7 (127.14.41.66:39791), cd8b60eb31174018a28f19837fad5007 (127.14.41.67:43651)
W20260925 10:10:46.572777 29401 leader_election.cc:341] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aee59a1978db4fa6aebf715bf7a8c9a7 (127.14.41.66:39791): Not found: Tablet not found: 42f21e0561474d83b04c118e54872835
W20260925 10:10:46.572883 29403 leader_election.cc:341] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd8b60eb31174018a28f19837fad5007 (127.14.41.67:43651): Not found: Tablet not found: 42f21e0561474d83b04c118e54872835
I20260925 10:10:46.572971 29403 leader_election.cc:302] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a47799273a7f445e9657d46fe248d153; no voters: aee59a1978db4fa6aebf715bf7a8c9a7, cd8b60eb31174018a28f19837fad5007
I20260925 10:10:46.573066 29797 raft_consensus.cc:2726] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:10:47.248349 29798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:47.248139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.65:46455)
0925 10:10:47.248268 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:47.248299 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260925 10:10:47.264544 29799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:47.264318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.66:44871)
0925 10:10:47.264445 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:47.264496 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260925 10:10:47.395664 29800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:47.395429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.67:37281)
0925 10:10:47.395570 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:47.395617 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260925 10:10:48.249110 29801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:48.248864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.65:42565)
0925 10:10:48.248994 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:48.249061 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260925 10:10:48.265326 29802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:48.265073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.66:43695)
0925 10:10:48.265228 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:48.265273 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
I20260925 10:10:48.396555 29803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:48.396220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.67:42703)
0925 10:10:48.396443 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:48.396486 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":62,"threads_started":1}
I20260925 10:10:49.249902 29804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:49.249663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.65:50371)
0925 10:10:49.249808 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:49.249853 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260925 10:10:49.266147 29805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:49.265908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.66:44077)
0925 10:10:49.266053 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:49.266095 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260925 10:10:49.327739 29806 raft_consensus.cc:482] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:10:49.327831 29806 raft_consensus.cc:504] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:49.328074 29806 leader_election.cc:288] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aee59a1978db4fa6aebf715bf7a8c9a7 (127.14.41.66:39791), cd8b60eb31174018a28f19837fad5007 (127.14.41.67:43651)
W20260925 10:10:49.328536 29403 leader_election.cc:341] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd8b60eb31174018a28f19837fad5007 (127.14.41.67:43651): Not found: Tablet not found: 42f21e0561474d83b04c118e54872835
W20260925 10:10:49.328677 29401 leader_election.cc:341] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aee59a1978db4fa6aebf715bf7a8c9a7 (127.14.41.66:39791): Not found: Tablet not found: 42f21e0561474d83b04c118e54872835
I20260925 10:10:49.328739 29401 leader_election.cc:302] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a47799273a7f445e9657d46fe248d153; no voters: aee59a1978db4fa6aebf715bf7a8c9a7, cd8b60eb31174018a28f19837fad5007
I20260925 10:10:49.328804 29806 raft_consensus.cc:2726] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:10:49.397472 29807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:49.397231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.67:56757)
0925 10:10:49.397382 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:49.397425 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
I20260925 10:10:50.250707 29808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:50.250446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.65:53135)
0925 10:10:50.250604 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:50.250653 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":73,"threads_started":1}
I20260925 10:10:50.267025 29809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:50.266791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.66:33959)
0925 10:10:50.266930 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:50.266969 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260925 10:10:50.398386 29810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:50.398108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.67:49289)
0925 10:10:50.398288 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:50.398335 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
I20260925 10:10:51.043354 29811 raft_consensus.cc:482] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:10:51.043474 29811 raft_consensus.cc:504] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:51.043669 29811 leader_election.cc:288] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aee59a1978db4fa6aebf715bf7a8c9a7 (127.14.41.66:39791), cd8b60eb31174018a28f19837fad5007 (127.14.41.67:43651)
W20260925 10:10:51.044087 29401 leader_election.cc:341] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aee59a1978db4fa6aebf715bf7a8c9a7 (127.14.41.66:39791): Not found: Tablet not found: 42f21e0561474d83b04c118e54872835
W20260925 10:10:51.044209 29403 leader_election.cc:341] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd8b60eb31174018a28f19837fad5007 (127.14.41.67:43651): Not found: Tablet not found: 42f21e0561474d83b04c118e54872835
I20260925 10:10:51.044246 29403 leader_election.cc:302] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a47799273a7f445e9657d46fe248d153; no voters: aee59a1978db4fa6aebf715bf7a8c9a7, cd8b60eb31174018a28f19837fad5007
I20260925 10:10:51.044338 29811 raft_consensus.cc:2726] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:10:51.251519 29812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:51.251271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.65:37419)
0925 10:10:51.251424 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:51.251470 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
I20260925 10:10:51.267750 29813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:51.267531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.66:59709)
0925 10:10:51.267671 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:51.267705 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260925 10:10:51.399212 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:51.398945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.67:42227)
0925 10:10:51.399126 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:51.399167 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":108,"threads_started":1}
I20260925 10:10:52.252364 29815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:52.252110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.65:43429)
0925 10:10:52.252254 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:52.252316 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260925 10:10:52.268424 29816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:52.268206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.66:52459)
0925 10:10:52.268337 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:52.268378 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260925 10:10:52.400071 29817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:52.399779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.67:56441)
0925 10:10:52.399968 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:52.400022 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":65,"threads_started":1}
I20260925 10:10:53.253104 29818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:53.252864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.65:59917)
0925 10:10:53.253022 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:53.253057 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260925 10:10:53.269354 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:53.268974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.66:56179)
0925 10:10:53.269260 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:53.269304 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":68,"threads_started":1}
I20260925 10:10:53.401018 29820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:53.400643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.67:41385)
0925 10:10:53.400859 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:53.400939 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":130,"threads_started":1}
I20260925 10:10:54.253901 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:54.253626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.65:47461)
0925 10:10:54.253802 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:54.253851 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":67,"threads_started":1}
I20260925 10:10:54.270047 29822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:54.269819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.66:45085)
0925 10:10:54.269960 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:54.269999 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260925 10:10:54.401793 29823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:54.401545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.67:59269)
0925 10:10:54.401706 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:54.401745 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260925 10:10:54.643802 29824 raft_consensus.cc:482] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:10:54.643898 29824 raft_consensus.cc:504] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee59a1978db4fa6aebf715bf7a8c9a7" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 39791 } } peers { permanent_uuid: "cd8b60eb31174018a28f19837fad5007" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 43651 } } peers { permanent_uuid: "a47799273a7f445e9657d46fe248d153" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 36629 } }
I20260925 10:10:54.644129 29824 leader_election.cc:288] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aee59a1978db4fa6aebf715bf7a8c9a7 (127.14.41.66:39791), cd8b60eb31174018a28f19837fad5007 (127.14.41.67:43651)
W20260925 10:10:54.644556 29403 leader_election.cc:341] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cd8b60eb31174018a28f19837fad5007 (127.14.41.67:43651): Not found: Tablet not found: 42f21e0561474d83b04c118e54872835
W20260925 10:10:54.644610 29401 leader_election.cc:341] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aee59a1978db4fa6aebf715bf7a8c9a7 (127.14.41.66:39791): Not found: Tablet not found: 42f21e0561474d83b04c118e54872835
I20260925 10:10:54.644637 29403 leader_election.cc:302] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a47799273a7f445e9657d46fe248d153; no voters: aee59a1978db4fa6aebf715bf7a8c9a7, cd8b60eb31174018a28f19837fad5007
I20260925 10:10:54.644711 29824 raft_consensus.cc:2726] T 42f21e0561474d83b04c118e54872835 P a47799273a7f445e9657d46fe248d153 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:10:55.254812 29825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:55.254543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.65:39807)
0925 10:10:55.254713 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:55.254760 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":68,"threads_started":1}
I20260925 10:10:55.270862 29826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:55.270634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.66:60527)
0925 10:10:55.270772 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:55.270812 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260925 10:10:55.402488 29827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:55.402272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.67:46115)
0925 10:10:55.402399 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:55.402442 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260925 10:10:56.255548 29828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:56.255299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.65:36375)
0925 10:10:56.255458 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:56.255500 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
I20260925 10:10:56.271625 29829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:56.271388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.66:55707)
0925 10:10:56.271537 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:56.271578 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260925 10:10:56.403183 29830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:56.402952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.67:60565)
0925 10:10:56.403092 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:56.403134 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260925 10:10:57.256387 29831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:57.256146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.65:52633)
0925 10:10:57.256296 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:57.256339 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260925 10:10:57.272457 29832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:57.272195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.66:34385)
0925 10:10:57.272359 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:57.272405 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
I20260925 10:10:57.404089 29833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:57.403773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.67:60853)
0925 10:10:57.403996 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:57.404039 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":79,"threads_started":1}
I20260925 10:10:58.257158 29834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:58.256952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.65:52205)
0925 10:10:58.257074 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:58.257110 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260925 10:10:58.273361 29835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:58.273098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.66:56045)
0925 10:10:58.273278 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:58.273315 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":111,"threads_started":1}
I20260925 10:10:58.404973 29836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:58.404727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:41135 (local address 127.14.41.67:47661)
0925 10:10:58.404874 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:58.404923 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260925 10:10:58.405516 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 29355
I20260925 10:10:58.410509 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 29517
I20260925 10:10:58.415120 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 29651
2026-09-25T10:10:58Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18312 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:10:58 dist-test-slave-s6tm krb5kdc[29839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:10:58 dist-test-slave-s6tm krb5kdc[29839](info): set up 2 sockets
Sep 25 10:10:58 dist-test-slave-s6tm krb5kdc[29839](info): commencing operation
krb5kdc: starting...
W20260925 10:10:59.461681 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:10:59 dist-test-slave-s6tm krb5kdc[29839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:10:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:10:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:10:59.590692 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:32949
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:32949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:10:59.676117 29852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:59.676350 29852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:59.676379 29852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:59.677675 29852 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 10:10:59.677731 29852 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 10:10:59.677744 29852 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 10:10:59.677755 29852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:10:59.677768 29852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:59.677824 29852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:59.677842 29852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:10:59.677873 29852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:10:59.679463 29852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:32949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:32949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:59.679754 29852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:59.680042 29852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:59.685992 29857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:59.686118 29860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:59.686012 29852 server_base.cc:1034] running on GCE node
W20260925 10:10:59.686228 29858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:59.686645 29852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:59.687029 29852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:59.688179 29852 hybrid_clock.cc:648] HybridClock initialized: now 1790331059688171 us; error 55 us; skew 500 ppm
Sep 25 10:10:59 dist-test-slave-s6tm krb5kdc[29839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331059, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:59.690939 29852 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:10:59.691461 29852 webserver.cc:460] Webserver started at http://127.14.41.126:39877/ using document root <none> and password file <none>
I20260925 10:10:59.691696 29852 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:59.691767 29852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:59.691963 29852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:10:59.693181 29852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "a6ca323bab744bcd93cef175968c3c17"
format_stamp: "Formatted at 2026-09-25 10:10:59 on dist-test-slave-s6tm"
server_key: "1c58dfb79a831334b621f512fd1771d6"
server_key_iv: "0eafa4e6a160e55f9c60b1a3f663f773"
server_key_version: "encryptionkey@0"
I20260925 10:10:59.693573 29852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "a6ca323bab744bcd93cef175968c3c17"
format_stamp: "Formatted at 2026-09-25 10:10:59 on dist-test-slave-s6tm"
server_key: "1c58dfb79a831334b621f512fd1771d6"
server_key_iv: "0eafa4e6a160e55f9c60b1a3f663f773"
server_key_version: "encryptionkey@0"
I20260925 10:10:59.695029 29852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:10:59.695905 29867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:59.696098 29852 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:10:59.696202 29852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "a6ca323bab744bcd93cef175968c3c17"
format_stamp: "Formatted at 2026-09-25 10:10:59 on dist-test-slave-s6tm"
server_key: "1c58dfb79a831334b621f512fd1771d6"
server_key_iv: "0eafa4e6a160e55f9c60b1a3f663f773"
server_key_version: "encryptionkey@0"
I20260925 10:10:59.696287 29852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:59.719108 29852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:59.722162 29852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:59.722280 29852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:59.726696 29852 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:32949
I20260925 10:10:59.726730 29919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:32949 every 8 connection(s)
I20260925 10:10:59.727126 29852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:10:59.728173 29920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:10:59.731123 29920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17: Bootstrap starting.
I20260925 10:10:59.731798 29920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17: Neither blocks nor log segments found. Creating new log.
I20260925 10:10:59.732174 29920 log.cc:824] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17: Log is configured to *not* fsync() on all Append() calls
I20260925 10:10:59.733038 29920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17: No bootstrap required, opened a new log
I20260925 10:10:59.734833 29920 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ca323bab744bcd93cef175968c3c17" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 32949 } }
I20260925 10:10:59.735044 29920 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:10:59.735085 29920 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6ca323bab744bcd93cef175968c3c17, State: Initialized, Role: FOLLOWER
I20260925 10:10:59.735273 29920 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ca323bab744bcd93cef175968c3c17" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 32949 } }
I20260925 10:10:59.735344 29920 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:10:59.735373 29920 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:10:59.735456 29920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:10:59.736259 29920 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ca323bab744bcd93cef175968c3c17" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 32949 } }
I20260925 10:10:59.736388 29920 leader_election.cc:302] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6ca323bab744bcd93cef175968c3c17; no voters:
I20260925 10:10:59.736460 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 29852
I20260925 10:10:59.736574 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:10:59.736605 29920 leader_election.cc:288] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:10:59.736635 29923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:10:59.736780 29923 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 1 LEADER]: Becoming Leader. State: Replica: a6ca323bab744bcd93cef175968c3c17, State: Running, Role: LEADER
I20260925 10:10:59.736860 14501 external_mini_cluster.cc:1428] Setting key 3672f59db0a9391e9c0bdf38d73d5bfc
I20260925 10:10:59.736887 29920 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:10:59.736940 29923 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ca323bab744bcd93cef175968c3c17" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 32949 } }
I20260925 10:10:59.738014 29924 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6ca323bab744bcd93cef175968c3c17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ca323bab744bcd93cef175968c3c17" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 32949 } } }
I20260925 10:10:59.738125 29924 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [sys.catalog]: This master's current role is: LEADER
I20260925 10:10:59.738444 29925 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6ca323bab744bcd93cef175968c3c17. Latest consensus state: current_term: 1 leader_uuid: "a6ca323bab744bcd93cef175968c3c17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ca323bab744bcd93cef175968c3c17" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 32949 } } }
I20260925 10:10:59.738648 29939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:10:59.739104 29925 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [sys.catalog]: This master's current role is: LEADER
I20260925 10:10:59.739528 29939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:10:59.741178 29939 catalog_manager.cc:1347] Generated new cluster ID: 74d160872e6d4e1f891407efe9bd4293
I20260925 10:10:59.741235 29939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:10:59 dist-test-slave-s6tm krb5kdc[29839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:10:59.750694 29932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:59.737554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39974 (local address 127.14.41.126:32949)
0925 10:10:59.737747 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:59.737761 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:59.737806 (+ 45us) server_negotiation.cc:407] Connection header received
0925 10:10:59.737954 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:59.737964 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:59.738035 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:59.739046 (+ 1011us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:59.739185 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:59.740800 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:59.741938 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:59.742648 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:59.742781 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:59.745931 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:59.745936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:59.745937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:59.745951 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:59.748834 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:59.749338 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:59.749341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:59.749342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:59.749459 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:59.749688 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:59.749689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:59.749689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:59.750021 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:59.750150 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:59.750321 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 10:10:59.750465 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":107,"threads_started":1}
I20260925 10:10:59.753793 29939 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:10:59.754397 29939 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:10:59.762432 29939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17: Generated new TSK 0
I20260925 10:10:59.762665 29939 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
I20260925 10:10:59.804418 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:32949
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:10:59.894516 29946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:10:59.894729 29946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:10:59.894754 29946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:10:59.896008 29946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:10:59.896054 29946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:10:59.896081 29946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:10:59.897591 29946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:10:59.897854 29946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:10:59.898114 29946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:10:59.904778 29951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:59.904843 29952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:10:59.904871 29954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:10:59.905125 29946 server_base.cc:1034] running on GCE node
I20260925 10:10:59.905351 29946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:10:59.905618 29946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:10:59.906797 29946 hybrid_clock.cc:648] HybridClock initialized: now 1790331059906781 us; error 33 us; skew 500 ppm
Sep 25 10:10:59 dist-test-slave-s6tm krb5kdc[29839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331059, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:10:59.909658 29946 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:10:59.910175 29946 webserver.cc:460] Webserver started at http://127.14.41.65:40233/ using document root <none> and password file <none>
I20260925 10:10:59.910475 29946 fs_manager.cc:359] Metadata directory not provided
I20260925 10:10:59.910545 29946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:10:59.910684 29946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:10:59.911758 29946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "19a9b1daf7c443a4acb86a081b74adde"
format_stamp: "Formatted at 2026-09-25 10:10:59 on dist-test-slave-s6tm"
server_key: "6ebde7883f7cedacee793c1e7e115cf6"
server_key_iv: "0f8bb9acfd7df3d2dd0e353e20c32653"
server_key_version: "encryptionkey@0"
I20260925 10:10:59.912164 29946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "19a9b1daf7c443a4acb86a081b74adde"
format_stamp: "Formatted at 2026-09-25 10:10:59 on dist-test-slave-s6tm"
server_key: "6ebde7883f7cedacee793c1e7e115cf6"
server_key_iv: "0f8bb9acfd7df3d2dd0e353e20c32653"
server_key_version: "encryptionkey@0"
I20260925 10:10:59.913625 29946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:10:59.914422 29961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:59.914621 29946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:10:59.914724 29946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "19a9b1daf7c443a4acb86a081b74adde"
format_stamp: "Formatted at 2026-09-25 10:10:59 on dist-test-slave-s6tm"
server_key: "6ebde7883f7cedacee793c1e7e115cf6"
server_key_iv: "0f8bb9acfd7df3d2dd0e353e20c32653"
server_key_version: "encryptionkey@0"
I20260925 10:10:59.914837 29946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:10:59.936957 29946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:10:59.939489 29946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:10:59.939668 29946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:10:59.939975 29946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:10:59.940335 29946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:10:59.940395 29946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:59.940446 29946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:10:59.940469 29946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:10:59.946545 29946 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:43961
I20260925 10:10:59.946594 30074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:43961 every 8 connection(s)
I20260925 10:10:59.946920 29946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:10:59.951153 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 29946
I20260925 10:10:59.951279 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:10:59.951507 14501 external_mini_cluster.cc:1428] Setting key 4497cda21556c786c4531634543b76dc
Sep 25 10:10:59 dist-test-slave-s6tm krb5kdc[29839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331059, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:10:59.957943 29932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:59.947658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:56275 (local address 127.14.41.126:32949)
0925 10:10:59.947733 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 10:10:59.947736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:10:59.948022 (+ 286us) server_negotiation.cc:407] Connection header received
0925 10:10:59.948575 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:10:59.948576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:10:59.948592 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:10:59.948652 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:10:59.949624 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:59.950192 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:59.951175 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:10:59.951552 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:10:59.951632 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:59.954495 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:10:59.954497 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:10:59.954497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:10:59.954507 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:10:59.956368 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:59.956823 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:59.956824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:59.956825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:59.956882 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:10:59.957163 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:10:59.957164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:10:59.957164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:10:59.957289 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:10:59.957459 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 10:10:59.957748 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 10:10:59.957839 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:10:59.958164 30077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:10:59.947652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.65:56275)
0925 10:10:59.947864 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 10:10:59.947903 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:10:59.948475 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:10:59.948738 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:10:59.948741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:10:59.949337 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:10:59.949541 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:59.949546 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:59.950276 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:59.950277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:59.951087 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:10:59.951089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:10:59.951711 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:10:59.951720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:10:59.951846 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:10:59.952613 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:10:59.952624 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:10:59.954389 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:10:59.956454 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:59.956461 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:59.956463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:59.956688 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:59.956960 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:10:59.956961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:10:59.956962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:10:59.957083 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:10:59.957450 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:10:59.957458 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:10:59.957602 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 10:10:59.957714 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 10:10:59.957923 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":127,"threads_started":1}
I20260925 10:10:59.958747 30075 heartbeater.cc:347] Connected to a master server at 127.14.41.126:32949
I20260925 10:10:59.958868 30075 heartbeater.cc:464] Registering TS with master...
I20260925 10:10:59.959108 30075 heartbeater.cc:511] Master 127.14.41.126:32949 requested a full tablet report, sending...
I20260925 10:10:59.959718 29884 ts_manager.cc:194] Registered new tserver with Master: 19a9b1daf7c443a4acb86a081b74adde (127.14.41.65:43961)
I20260925 10:10:59.960408 29884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:56275
WARNING: no policy specified for kudu/127.14.41.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
I20260925 10:10:59.983889 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:32949
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:11:00.068361 30080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:11:00.068584 30080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:11:00.068645 30080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:11:00.070175 30080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:11:00.070241 30080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:11:00.070272 30080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:11:00.071866 30080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:11:00.072108 30080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:11:00.072319 30080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:11:00.077548 30085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:00.077474 30088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:00.077544 30086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:00.077872 30080 server_base.cc:1034] running on GCE node
I20260925 10:11:00.078120 30080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:11:00.078392 30080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:11:00.079569 30080 hybrid_clock.cc:648] HybridClock initialized: now 1790331060079548 us; error 47 us; skew 500 ppm
Sep 25 10:11:00 dist-test-slave-s6tm krb5kdc[29839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331060, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:11:00.082243 30080 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:11:00.082743 30080 webserver.cc:460] Webserver started at http://127.14.41.66:40639/ using document root <none> and password file <none>
I20260925 10:11:00.082996 30080 fs_manager.cc:359] Metadata directory not provided
I20260925 10:11:00.083066 30080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:11:00.083214 30080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:11:00.084455 30080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "c9bbc45632c446d9a914a38f78c6c771"
format_stamp: "Formatted at 2026-09-25 10:11:00 on dist-test-slave-s6tm"
server_key: "e56118a112a827f1851c9944df779db7"
server_key_iv: "0296d7f8cb59eeca3c21a6e01e5d70ea"
server_key_version: "encryptionkey@0"
I20260925 10:11:00.084836 30080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "c9bbc45632c446d9a914a38f78c6c771"
format_stamp: "Formatted at 2026-09-25 10:11:00 on dist-test-slave-s6tm"
server_key: "e56118a112a827f1851c9944df779db7"
server_key_iv: "0296d7f8cb59eeca3c21a6e01e5d70ea"
server_key_version: "encryptionkey@0"
I20260925 10:11:00.086220 30080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:11:00.087049 30095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:00.087237 30080 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:11:00.087393 30080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "c9bbc45632c446d9a914a38f78c6c771"
format_stamp: "Formatted at 2026-09-25 10:11:00 on dist-test-slave-s6tm"
server_key: "e56118a112a827f1851c9944df779db7"
server_key_iv: "0296d7f8cb59eeca3c21a6e01e5d70ea"
server_key_version: "encryptionkey@0"
I20260925 10:11:00.087482 30080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:11:00.114706 30080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:11:00.116947 30080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:11:00.117120 30080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:11:00.117389 30080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:11:00.117789 30080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:11:00.117843 30080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:00.117879 30080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:11:00.117921 30080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:00.124162 30080 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:40815
I20260925 10:11:00.124236 30208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:40815 every 8 connection(s)
I20260925 10:11:00.124635 30080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:11:00.129642 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 30080
I20260925 10:11:00.129766 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:11:00.130013 14501 external_mini_cluster.cc:1428] Setting key cf4b328b38820ddbaf36b36ef55db79d
Sep 25 10:11:00 dist-test-slave-s6tm krb5kdc[29839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331060, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:11:00.135553 29932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:00.125469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:60967 (local address 127.14.41.126:32949)
0925 10:11:00.125541 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:00.125543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:00.125745 (+ 202us) server_negotiation.cc:407] Connection header received
0925 10:11:00.126247 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:00.126248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:00.126258 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:00.126312 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:11:00.127451 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.127969 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.128932 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.129115 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.129170 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:00.132010 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:11:00.132011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:11:00.132012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:11:00.132019 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:11:00.133922 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:00.134329 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:00.134330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:00.134331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:00.134400 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:00.134685 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:00.134686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:00.134687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:00.134851 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:11:00.134942 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:00.135326 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 10:11:00.135427 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:11:00.135562 30211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:00.125441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.66:60967)
0925 10:11:00.125617 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:00.125660 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:00.126165 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:00.126378 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:00.126382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:00.126992 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:11:00.127284 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:00.127291 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.128050 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.128051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:00.128854 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:00.128856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.129189 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.129190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:00.129290 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:00.129863 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:11:00.129875 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:11:00.131842 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:11:00.134013 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:11:00.134019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:11:00.134021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:11:00.134242 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:11:00.134490 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:11:00.134491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:11:00.134491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:11:00.134601 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:11:00.134930 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:11:00.134935 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:11:00.135165 (+ 230us) client_negotiation.cc:769] Sending connection context
0925 10:11:00.135256 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 10:11:00.135393 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260925 10:11:00.136185 30209 heartbeater.cc:347] Connected to a master server at 127.14.41.126:32949
I20260925 10:11:00.136300 30209 heartbeater.cc:464] Registering TS with master...
I20260925 10:11:00.136509 30209 heartbeater.cc:511] Master 127.14.41.126:32949 requested a full tablet report, sending...
I20260925 10:11:00.137131 29884 ts_manager.cc:194] Registered new tserver with Master: c9bbc45632c446d9a914a38f78c6c771 (127.14.41.66:40815)
I20260925 10:11:00.137777 29884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:60967
WARNING: no policy specified for kudu/127.14.41.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
I20260925 10:11:00.159129 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:32949
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:11:00.243686 30214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:11:00.243906 30214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:11:00.243989 30214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:11:00.245200 30214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:11:00.245246 30214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:11:00.245275 30214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:11:00.246788 30214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:11:00.247045 30214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:11:00.247277 30214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:11:00.253235 30219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:00.253211 30220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:00.253254 30214 server_base.cc:1034] running on GCE node
W20260925 10:11:00.253216 30222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:00.253733 30214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:11:00.254000 30214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:11:00.255152 30214 hybrid_clock.cc:648] HybridClock initialized: now 1790331060255138 us; error 30 us; skew 500 ppm
Sep 25 10:11:00 dist-test-slave-s6tm krb5kdc[29839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331060, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:11:00.257817 30214 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:11:00.258396 30214 webserver.cc:460] Webserver started at http://127.14.41.67:36917/ using document root <none> and password file <none>
I20260925 10:11:00.258659 30214 fs_manager.cc:359] Metadata directory not provided
I20260925 10:11:00.258729 30214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:11:00.258881 30214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:11:00.260030 30214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "e766569a581845a49536bca5db88c659"
format_stamp: "Formatted at 2026-09-25 10:11:00 on dist-test-slave-s6tm"
server_key: "9aca95e3863e4959ff33d24df752a428"
server_key_iv: "edb7cfdaed257442fb8b2ec35b5b8fae"
server_key_version: "encryptionkey@0"
I20260925 10:11:00.260408 30214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "e766569a581845a49536bca5db88c659"
format_stamp: "Formatted at 2026-09-25 10:11:00 on dist-test-slave-s6tm"
server_key: "9aca95e3863e4959ff33d24df752a428"
server_key_iv: "edb7cfdaed257442fb8b2ec35b5b8fae"
server_key_version: "encryptionkey@0"
I20260925 10:11:00.261791 30214 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:11:00.262667 30229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:00.262849 30214 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:11:00.262946 30214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "e766569a581845a49536bca5db88c659"
format_stamp: "Formatted at 2026-09-25 10:11:00 on dist-test-slave-s6tm"
server_key: "9aca95e3863e4959ff33d24df752a428"
server_key_iv: "edb7cfdaed257442fb8b2ec35b5b8fae"
server_key_version: "encryptionkey@0"
I20260925 10:11:00.263031 30214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:11:00.278555 30214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:11:00.280849 30214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:11:00.281024 30214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:11:00.281277 30214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:11:00.281612 30214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:11:00.281669 30214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:00.281713 30214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:11:00.281744 30214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:00.287887 30214 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:46465
I20260925 10:11:00.287946 30342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:46465 every 8 connection(s)
I20260925 10:11:00.288302 30214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:11:00.294572 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 30214
I20260925 10:11:00.294674 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:11:00.294906 14501 external_mini_cluster.cc:1428] Setting key b0e0bfc9ac146373d519f867dd788e02
Sep 25 10:11:00 dist-test-slave-s6tm krb5kdc[29839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331060, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:11:00.298568 29932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:00.289097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:42973 (local address 127.14.41.126:32949)
0925 10:11:00.289169 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:00.289171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:00.289367 (+ 196us) server_negotiation.cc:407] Connection header received
0925 10:11:00.289854 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:00.289855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:00.289866 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:00.289929 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:11:00.291026 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.291551 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.292835 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.293030 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.293090 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:00.295740 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:11:00.295742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:11:00.295742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:11:00.295750 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:11:00.297062 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:00.297543 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:00.297545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:00.297545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:00.297610 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:00.297875 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:00.297876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:00.297877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:00.297996 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:11:00.298076 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:00.298359 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 10:11:00.298448 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:11:00.298786 30345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:00.289052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.67:42973)
0925 10:11:00.289233 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:00.289277 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:00.289766 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:00.290006 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:00.290010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:00.290705 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:11:00.290920 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:00.290926 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.291690 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.291691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:00.292746 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:00.292749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.293110 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.293111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:00.293218 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:00.293903 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:11:00.293912 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:11:00.295510 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:11:00.297164 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:11:00.297171 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:11:00.297173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:11:00.297444 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:11:00.297680 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:11:00.297682 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:11:00.297682 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:11:00.297804 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:11:00.298067 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:11:00.298075 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:11:00.298222 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 10:11:00.298337 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 10:11:00.298502 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260925 10:11:00.299243 30343 heartbeater.cc:347] Connected to a master server at 127.14.41.126:32949
I20260925 10:11:00.299369 30343 heartbeater.cc:464] Registering TS with master...
I20260925 10:11:00.299561 30343 heartbeater.cc:511] Master 127.14.41.126:32949 requested a full tablet report, sending...
I20260925 10:11:00.300052 29884 ts_manager.cc:194] Registered new tserver with Master: e766569a581845a49536bca5db88c659 (127.14.41.67:46465)
I20260925 10:11:00.300412 29884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:42973
I20260925 10:11:00.306699 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:11:00.314438 29932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:00.308155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39976 (local address 127.14.41.126:32949)
0925 10:11:00.308225 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:00.308228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:00.308337 (+ 109us) server_negotiation.cc:407] Connection header received
0925 10:11:00.308386 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:00.308386 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:00.308396 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:00.308442 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:11:00.308971 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.309469 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.310163 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.310342 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.310393 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:00.311524 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:11:00.311526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:11:00.311526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:11:00.311532 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:11:00.312998 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:00.313361 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:00.313361 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:00.313362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:00.313412 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:00.313581 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:00.313581 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:00.313581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:00.313722 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:11:00.313804 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:00.314041 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 10:11:00.314234 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:11:00.316174 29884 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:11:00.318037 29884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:11:00.325207 30355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:00.320752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:43961 (local address 127.0.0.1:58618)
0925 10:11:00.321244 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:00.321255 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:00.321321 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:00.321724 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:00.321726 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:00.321727 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:11:00.321772 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:00.321773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.322950 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.322951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:00.324109 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:00.324112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.324874 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.324875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:00.324955 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:00.324971 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 10:11:00.325070 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 10:11:00.325126 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":37,"thread_start_us":128,"threads_started":1}
I20260925 10:11:00.325213 30358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:00.321074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58618 (local address 127.14.41.65:43961)
0925 10:11:00.321260 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:00.321264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:00.321309 (+ 45us) server_negotiation.cc:407] Connection header received
0925 10:11:00.321594 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:00.321597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:00.321614 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:00.321721 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:11:00.321838 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.322859 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.324230 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.324776 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.324872 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:00.324939 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:00.325088 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 10:11:00.325127 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260925 10:11:00.325343 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:00.320730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:40815 (local address 127.0.0.1:39854)
0925 10:11:00.320838 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:00.320876 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:00.320944 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:00.321328 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:00.321331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:00.321334 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:11:00.321402 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:00.321405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.322742 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.322742 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:00.324109 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:00.324112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.325106 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.325107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:00.325178 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:00.325181 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:11:00.325228 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 10:11:00.325265 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260925 10:11:00.325623 30356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:00.320880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39854 (local address 127.14.41.66:40815)
0925 10:11:00.321064 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:00.321070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:00.321080 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:11:00.321118 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:00.321121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:00.321135 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:00.321233 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:11:00.321494 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.322645 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.324203 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.324926 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.325010 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:00.325071 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:00.325503 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 10:11:00.325548 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
I20260925 10:11:00.326326 30357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:00.320980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:46465 (local address 127.0.0.1:34786)
0925 10:11:00.321400 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:00.321410 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:00.321453 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:00.321728 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:00.321728 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:00.321729 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:11:00.321774 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:00.321776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.323910 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.323911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:00.324997 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:00.325000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.326093 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.326094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:00.326155 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:00.326159 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:11:00.326215 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 10:11:00.326254 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":63,"threads_started":1}
I20260925 10:11:00.326346 30143 tablet_service.cc:1467] Processing CreateTablet for tablet d3213dccd4544ea5b56750b42fc98185 (DEFAULT_TABLE table=test-table [id=67960c314ec24155bfa0c68e4694bc3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:11:00.326364 30009 tablet_service.cc:1467] Processing CreateTablet for tablet d3213dccd4544ea5b56750b42fc98185 (DEFAULT_TABLE table=test-table [id=67960c314ec24155bfa0c68e4694bc3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:11:00.326548 30359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:00.321172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34786 (local address 127.14.41.67:46465)
0925 10:11:00.321466 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:00.321470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:00.321480 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:11:00.321524 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:00.321526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:00.321574 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:00.321647 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:11:00.322956 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.323805 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.325085 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.325938 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.326007 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:00.326060 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:00.326249 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 10:11:00.326295 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":83,"threads_started":1}
I20260925 10:11:00.326732 30009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3213dccd4544ea5b56750b42fc98185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:11:00.326733 30143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3213dccd4544ea5b56750b42fc98185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:11:00.327154 30277 tablet_service.cc:1467] Processing CreateTablet for tablet d3213dccd4544ea5b56750b42fc98185 (DEFAULT_TABLE table=test-table [id=67960c314ec24155bfa0c68e4694bc3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:11:00.327453 30277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3213dccd4544ea5b56750b42fc98185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:11:00.329494 30360 tablet_bootstrap.cc:492] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771: Bootstrap starting.
I20260925 10:11:00.330108 30360 tablet_bootstrap.cc:654] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771: Neither blocks nor log segments found. Creating new log.
I20260925 10:11:00.330567 30361 tablet_bootstrap.cc:492] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde: Bootstrap starting.
I20260925 10:11:00.330626 30360 log.cc:824] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771: Log is configured to *not* fsync() on all Append() calls
I20260925 10:11:00.330837 30362 tablet_bootstrap.cc:492] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659: Bootstrap starting.
I20260925 10:11:00.331209 30361 tablet_bootstrap.cc:654] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde: Neither blocks nor log segments found. Creating new log.
I20260925 10:11:00.331458 30362 tablet_bootstrap.cc:654] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659: Neither blocks nor log segments found. Creating new log.
I20260925 10:11:00.331537 30360 tablet_bootstrap.cc:492] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771: No bootstrap required, opened a new log
I20260925 10:11:00.331611 30360 ts_tablet_manager.cc:1397] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:11:00.331801 30361 log.cc:824] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde: Log is configured to *not* fsync() on all Append() calls
I20260925 10:11:00.332044 30362 log.cc:824] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659: Log is configured to *not* fsync() on all Append() calls
I20260925 10:11:00.332672 30361 tablet_bootstrap.cc:492] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde: No bootstrap required, opened a new log
I20260925 10:11:00.332764 30361 ts_tablet_manager.cc:1397] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:11:00.332950 30362 tablet_bootstrap.cc:492] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659: No bootstrap required, opened a new log
I20260925 10:11:00.333026 30362 ts_tablet_manager.cc:1397] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:11:00.333840 30360 raft_consensus.cc:348] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } } peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } }
I20260925 10:11:00.334040 30360 raft_consensus.cc:374] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:11:00.334080 30360 raft_consensus.cc:729] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9bbc45632c446d9a914a38f78c6c771, State: Initialized, Role: FOLLOWER
I20260925 10:11:00.334194 30360 consensus_queue.cc:260] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } } peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } }
I20260925 10:11:00.334483 30360 ts_tablet_manager.cc:1428] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:11:00.334497 30209 heartbeater.cc:503] Master 127.14.41.126:32949 was elected leader, sending a full tablet report...
I20260925 10:11:00.334615 30361 raft_consensus.cc:348] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } } peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } }
I20260925 10:11:00.334851 30361 raft_consensus.cc:374] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:11:00.334940 30361 raft_consensus.cc:729] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a9b1daf7c443a4acb86a081b74adde, State: Initialized, Role: FOLLOWER
I20260925 10:11:00.335074 30361 consensus_queue.cc:260] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } } peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } }
I20260925 10:11:00.335067 30362 raft_consensus.cc:348] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } } peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } }
I20260925 10:11:00.335215 30362 raft_consensus.cc:374] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:11:00.335261 30362 raft_consensus.cc:729] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e766569a581845a49536bca5db88c659, State: Initialized, Role: FOLLOWER
I20260925 10:11:00.335402 30361 ts_tablet_manager.cc:1428] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 10:11:00.335389 30362 consensus_queue.cc:260] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } } peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } }
I20260925 10:11:00.335511 30075 heartbeater.cc:503] Master 127.14.41.126:32949 was elected leader, sending a full tablet report...
I20260925 10:11:00.335662 30362 ts_tablet_manager.cc:1428] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 10:11:00.335769 30343 heartbeater.cc:503] Master 127.14.41.126:32949 was elected leader, sending a full tablet report...
I20260925 10:11:00.351693 30368 raft_consensus.cc:482] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:11:00.351818 30368 raft_consensus.cc:504] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } } peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } }
I20260925 10:11:00.352444 30368 leader_election.cc:288] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9bbc45632c446d9a914a38f78c6c771 (127.14.41.66:40815), 19a9b1daf7c443a4acb86a081b74adde (127.14.41.65:43961)
I20260925 10:11:00.355296 30358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:00.352745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:55329 (local address 127.14.41.65:43961)
0925 10:11:00.352792 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:00.352794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:00.352802 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:11:00.352831 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:00.352831 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:00.352841 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:00.352899 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:11:00.353072 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.353593 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.354527 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.354923 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.354988 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:00.354995 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:00.355170 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 10:11:00.355221 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:11:00.355322 30345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:00.352599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:43961 (local address 127.14.41.67:55329)
0925 10:11:00.352659 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:00.352675 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:00.352726 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:00.352911 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:00.352913 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:00.352914 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:11:00.352994 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:00.352996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.353670 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.353671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:00.354439 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:00.354442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.354999 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.355000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:00.355095 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:00.355099 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:11:00.355153 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 10:11:00.355187 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 10:11:00.355372 30356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:00.352740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:51161 (local address 127.14.41.66:40815)
0925 10:11:00.352786 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:00.352788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:00.352843 (+ 55us) server_negotiation.cc:407] Connection header received
0925 10:11:00.352915 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:00.352915 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:00.352925 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:00.352983 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:11:00.353170 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.353646 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.354589 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.354995 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.355069 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:00.355078 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:00.355249 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 10:11:00.355301 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:11:00.355446 30369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:00.352599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:40815 (local address 127.14.41.67:51161)
0925 10:11:00.352780 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:00.352792 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:00.352856 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:00.352996 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:00.352997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:00.353002 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:11:00.353093 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:00.353094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.353728 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.353728 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:00.354465 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:00.354466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.355078 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.355079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:00.355157 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:00.355162 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:11:00.355219 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 10:11:00.355258 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":13,"thread_start_us":104,"threads_started":1}
I20260925 10:11:00.355798 30163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3213dccd4544ea5b56750b42fc98185" candidate_uuid: "e766569a581845a49536bca5db88c659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9bbc45632c446d9a914a38f78c6c771" is_pre_election: true
I20260925 10:11:00.356029 30163 raft_consensus.cc:2454] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e766569a581845a49536bca5db88c659 in term 0.
I20260925 10:11:00.356086 30029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3213dccd4544ea5b56750b42fc98185" candidate_uuid: "e766569a581845a49536bca5db88c659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a9b1daf7c443a4acb86a081b74adde" is_pre_election: true
I20260925 10:11:00.356240 30029 raft_consensus.cc:2454] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e766569a581845a49536bca5db88c659 in term 0.
I20260925 10:11:00.356262 30233 leader_election.cc:302] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9bbc45632c446d9a914a38f78c6c771, e766569a581845a49536bca5db88c659; no voters:
I20260925 10:11:00.356397 30368 raft_consensus.cc:2781] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:11:00.356473 30368 raft_consensus.cc:482] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:11:00.356546 30368 raft_consensus.cc:3037] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:11:00.357259 30368 raft_consensus.cc:504] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } } peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } }
I20260925 10:11:00.357416 30368 leader_election.cc:288] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [CANDIDATE]: Term 1 election: Requested vote from peers c9bbc45632c446d9a914a38f78c6c771 (127.14.41.66:40815), 19a9b1daf7c443a4acb86a081b74adde (127.14.41.65:43961)
I20260925 10:11:00.357606 30029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3213dccd4544ea5b56750b42fc98185" candidate_uuid: "e766569a581845a49536bca5db88c659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a9b1daf7c443a4acb86a081b74adde"
I20260925 10:11:00.357654 30163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3213dccd4544ea5b56750b42fc98185" candidate_uuid: "e766569a581845a49536bca5db88c659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9bbc45632c446d9a914a38f78c6c771"
I20260925 10:11:00.357693 30029 raft_consensus.cc:3037] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:11:00.357740 30163 raft_consensus.cc:3037] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:11:00.358409 30029 raft_consensus.cc:2454] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e766569a581845a49536bca5db88c659 in term 1.
I20260925 10:11:00.358569 30163 raft_consensus.cc:2454] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e766569a581845a49536bca5db88c659 in term 1.
I20260925 10:11:00.358605 30230 leader_election.cc:302] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a9b1daf7c443a4acb86a081b74adde, e766569a581845a49536bca5db88c659; no voters:
I20260925 10:11:00.358690 30368 raft_consensus.cc:2781] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:11:00.358847 30368 raft_consensus.cc:686] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [term 1 LEADER]: Becoming Leader. State: Replica: e766569a581845a49536bca5db88c659, State: Running, Role: LEADER
I20260925 10:11:00.358960 30368 consensus_queue.cc:237] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } } peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } }
I20260925 10:11:00.359869 29882 catalog_manager.cc:5495] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 reported cstate change: term changed from 0 to 1, leader changed from <none> to e766569a581845a49536bca5db88c659 (127.14.41.67). New cstate: current_term: 1 leader_uuid: "e766569a581845a49536bca5db88c659" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } health_report { overall_health: HEALTHY } } }
I20260925 10:11:00.374599 30359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:00.371778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34792 (local address 127.14.41.67:46465)
0925 10:11:00.371825 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:00.371828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:00.371837 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:11:00.371949 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:00.371950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:00.371974 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:00.372090 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:11:00.372242 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.372733 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.373542 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:00.373860 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:00.373990 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:00.374202 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:11:00.374313 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:11:00.374410 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:00.374456 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 10:11:00.374479 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 10:11:00.375214 30210 tablet.cc:2369] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:11:00.376784 30029 raft_consensus.cc:1261] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Refusing update from remote peer e766569a581845a49536bca5db88c659: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:11:00.377167 30163 raft_consensus.cc:1261] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771 [term 1 FOLLOWER]: Refusing update from remote peer e766569a581845a49536bca5db88c659: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:11:00.377298 30370 consensus_queue.cc:1035] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:11:00.377556 30368 consensus_queue.cc:1035] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:11:00.380013 30375 mvcc.cc:205] Tried to move back new op lower bound from 7333196023300190208 to 7333196023231299584. Current Snapshot: MvccSnapshot[applied={T|T < 7333196023300190208}]
I20260925 10:11:00.380795 30374 mvcc.cc:205] Tried to move back new op lower bound from 7333196023300190208 to 7333196023231299584. Current Snapshot: MvccSnapshot[applied={T|T < 7333196023300190208}]
I20260925 10:11:00.382333 29882 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:11:00.382421 29882 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:11:00.383203 29882 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17: Sending DeleteTablet for 3 replicas of tablet d3213dccd4544ea5b56750b42fc98185
I20260925 10:11:00.383564 30143 tablet_service.cc:1514] Processing DeleteTablet for tablet d3213dccd4544ea5b56750b42fc98185 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:11:00 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:39854
I20260925 10:11:00.383641 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 29946
I20260925 10:11:00.383617 30009 tablet_service.cc:1514] Processing DeleteTablet for tablet d3213dccd4544ea5b56750b42fc98185 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:11:00 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:58618
I20260925 10:11:00.384292 30277 tablet_service.cc:1514] Processing DeleteTablet for tablet d3213dccd4544ea5b56750b42fc98185 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:11:00 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:34786
I20260925 10:11:00.384529 30384 tablet_replica.cc:330] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659: stopping tablet replica
I20260925 10:11:00.384647 30384 raft_consensus.cc:2229] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:11:00.384784 30384 raft_consensus.cc:2258] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:11:00.385280 30384 ts_tablet_manager.cc:1905] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:11:00.387351 30384 ts_tablet_manager.cc:1918] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:11:00.387439 30384 log.cc:1196] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/d3213dccd4544ea5b56750b42fc98185
I20260925 10:11:00.387710 30384 ts_tablet_manager.cc:1939] T d3213dccd4544ea5b56750b42fc98185 P e766569a581845a49536bca5db88c659: Deleting consensus metadata
I20260925 10:11:00.388075 30383 tablet_replica.cc:330] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771: stopping tablet replica
I20260925 10:11:00.388209 29868 catalog_manager.cc:4841] TS e766569a581845a49536bca5db88c659 (127.14.41.67:46465): tablet d3213dccd4544ea5b56750b42fc98185 (table test-table [id=67960c314ec24155bfa0c68e4694bc3e]) successfully deleted
I20260925 10:11:00.388619 30383 raft_consensus.cc:2229] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:11:00.388753 30383 raft_consensus.cc:2258] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:11:00.389231 30383 ts_tablet_manager.cc:1905] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:11:00.390851 30383 ts_tablet_manager.cc:1918] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:11:00.390931 30383 log.cc:1196] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/d3213dccd4544ea5b56750b42fc98185
I20260925 10:11:00.391220 30383 ts_tablet_manager.cc:1939] T d3213dccd4544ea5b56750b42fc98185 P c9bbc45632c446d9a914a38f78c6c771: Deleting consensus metadata
W20260925 10:11:00.391423 29868 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.41.65:43961 (error 108)
I20260925 10:11:00.391610 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 30080
I20260925 10:11:00.392167 29871 catalog_manager.cc:4841] TS c9bbc45632c446d9a914a38f78c6c771 (127.14.41.66:40815): tablet d3213dccd4544ea5b56750b42fc98185 (table test-table [id=67960c314ec24155bfa0c68e4694bc3e]) successfully deleted
I20260925 10:11:00.392690 30357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:00.392539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:43961 (local address 127.0.0.1:58626)
0925 10:11:00.392606 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:00.392654 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.65:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 10:11:00.392807 29868 catalog_manager.cc:4584] TS 19a9b1daf7c443a4acb86a081b74adde (127.14.41.65:43961): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d3213dccd4544ea5b56750b42fc98185: Network error: Client connection negotiation failed: client connection to 127.14.41.65:43961: connect: Connection refused (error 111)
I20260925 10:11:00.397694 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 30214
I20260925 10:11:00.402696 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 29852
I20260925 10:11:20.408017 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:32949
--webserver_interface=127.14.41.126
--webserver_port=39877
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:32949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:11:20.491565 30386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:11:20.491873 30386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:11:20.491917 30386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:11:20.493250 30386 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 10:11:20.493338 30386 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 10:11:20.493360 30386 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 10:11:20.493378 30386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:11:20.493407 30386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:11:20.493438 30386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:11:20.493461 30386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:11:20.493491 30386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:11:20.495096 30386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:32949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:32949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:11:20.495406 30386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:11:20.495649 30386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:11:20.505749 30393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:20.505749 30392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:20.505838 30386 server_base.cc:1034] running on GCE node
W20260925 10:11:20.505756 30395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:20.506469 30386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:11:20.506837 30386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:11:20.508026 30386 hybrid_clock.cc:648] HybridClock initialized: now 1790331080508001 us; error 43 us; skew 500 ppm
Sep 25 10:11:20 dist-test-slave-s6tm krb5kdc[29839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331080, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:11:20.517275 30386 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:11:20.517846 30386 webserver.cc:460] Webserver started at http://127.14.41.126:39877/ using document root <none> and password file <none>
I20260925 10:11:20.518085 30386 fs_manager.cc:359] Metadata directory not provided
I20260925 10:11:20.518143 30386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:11:20.519644 30386 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:11:20.520483 30402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:20.520680 30386 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:11:20.520782 30386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "a6ca323bab744bcd93cef175968c3c17"
format_stamp: "Formatted at 2026-09-25 10:10:59 on dist-test-slave-s6tm"
server_key: "1c58dfb79a831334b621f512fd1771d6"
server_key_iv: "0eafa4e6a160e55f9c60b1a3f663f773"
server_key_version: "encryptionkey@0"
I20260925 10:11:20.521286 30386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:11:20.533713 30386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:11:20.536566 30386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:11:20.536739 30386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:11:20.540793 30386 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:32949
I20260925 10:11:20.540848 30454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:32949 every 8 connection(s)
I20260925 10:11:20.541270 30386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:11:20.543036 30455 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:11:20.543749 30455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17: Bootstrap starting.
I20260925 10:11:20.544230 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 30386
I20260925 10:11:20.544595 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:43961
--local_ip_for_outbound_sockets=127.14.41.65
--tserver_master_addrs=127.14.41.126:32949
--webserver_port=40233
--webserver_interface=127.14.41.65
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:11:20.553733 30455 log.cc:824] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17: Log is configured to *not* fsync() on all Append() calls
I20260925 10:11:20.556089 30455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:11:20.556499 30455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17: Bootstrap complete.
I20260925 10:11:20.559023 30455 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ca323bab744bcd93cef175968c3c17" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 32949 } }
I20260925 10:11:20.559278 30455 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6ca323bab744bcd93cef175968c3c17, State: Initialized, Role: FOLLOWER
I20260925 10:11:20.559475 30455 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ca323bab744bcd93cef175968c3c17" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 32949 } }
I20260925 10:11:20.559556 30455 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:11:20.559630 30455 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:11:20.559685 30455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:11:20.560884 30455 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ca323bab744bcd93cef175968c3c17" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 32949 } }
I20260925 10:11:20.560983 30455 leader_election.cc:302] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6ca323bab744bcd93cef175968c3c17; no voters:
I20260925 10:11:20.561228 30455 leader_election.cc:288] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:11:20.561270 30459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:11:20.561393 30459 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 2 LEADER]: Becoming Leader. State: Replica: a6ca323bab744bcd93cef175968c3c17, State: Running, Role: LEADER
I20260925 10:11:20.561487 30459 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ca323bab744bcd93cef175968c3c17" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 32949 } }
I20260925 10:11:20.561540 30455 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:11:20.561775 30459 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6ca323bab744bcd93cef175968c3c17. Latest consensus state: current_term: 2 leader_uuid: "a6ca323bab744bcd93cef175968c3c17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ca323bab744bcd93cef175968c3c17" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 32949 } } }
I20260925 10:11:20.561863 30459 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [sys.catalog]: This master's current role is: LEADER
I20260925 10:11:20.561784 30460 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a6ca323bab744bcd93cef175968c3c17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ca323bab744bcd93cef175968c3c17" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 32949 } } }
I20260925 10:11:20.562034 30460 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [sys.catalog]: This master's current role is: LEADER
I20260925 10:11:20.562071 30463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:11:20.563011 30463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:11:20.563126 30463 catalog_manager.cc:1259] Loaded cluster ID: 74d160872e6d4e1f891407efe9bd4293
I20260925 10:11:20.563156 30463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:11:20.564344 30463 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:11:20.564555 30463 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17: Loaded TSK: 0
I20260925 10:11:20.577873 30463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17: Generated new TSK 1
I20260925 10:11:20.578004 30463 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:11:20.643033 30457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:11:20.643260 30457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:11:20.643286 30457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:11:20.644622 30457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:11:20.644668 30457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:11:20.644699 30457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:11:20.646234 30457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:43961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=40233
--tserver_master_addrs=127.14.41.126:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:11:20.646485 30457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:11:20.646755 30457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:11:20.653549 30485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:20.653563 30482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:20.653599 30483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:20.653781 30457 server_base.cc:1034] running on GCE node
I20260925 10:11:20.654093 30457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:11:20.654503 30457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:11:20.655699 30457 hybrid_clock.cc:648] HybridClock initialized: now 1790331080655680 us; error 28 us; skew 500 ppm
Sep 25 10:11:20 dist-test-slave-s6tm krb5kdc[29839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331080, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:11:20.658519 30457 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:11:20.659070 30457 webserver.cc:460] Webserver started at http://127.14.41.65:40233/ using document root <none> and password file <none>
I20260925 10:11:20.659317 30457 fs_manager.cc:359] Metadata directory not provided
I20260925 10:11:20.659379 30457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:11:20.660967 30457 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:11:20.661747 30492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:20.661937 30457 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:11:20.662029 30457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "19a9b1daf7c443a4acb86a081b74adde"
format_stamp: "Formatted at 2026-09-25 10:10:59 on dist-test-slave-s6tm"
server_key: "6ebde7883f7cedacee793c1e7e115cf6"
server_key_iv: "0f8bb9acfd7df3d2dd0e353e20c32653"
server_key_version: "encryptionkey@0"
I20260925 10:11:20.662492 30457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:11:20.677557 30457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:11:20.680510 30457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:11:20.680704 30457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:11:20.680967 30457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:11:20.681454 30499 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 10:11:20.683439 30457 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 10:11:20.683537 30457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 10:11:20.683589 30457 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 10:11:20.684412 30457 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 10:11:20.684473 30457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260925 10:11:20.684556 30499 tablet_bootstrap.cc:492] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde: Bootstrap starting.
I20260925 10:11:20.692628 30457 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:43961
I20260925 10:11:20.692742 30606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:43961 every 8 connection(s)
I20260925 10:11:20.693212 30457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:11:20.697176 30499 log.cc:824] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde: Log is configured to *not* fsync() on all Append() calls
I20260925 10:11:20.699841 30499 tablet_bootstrap.cc:492] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:11:20.700189 30499 tablet_bootstrap.cc:492] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde: Bootstrap complete.
I20260925 10:11:20.700647 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 30457
I20260925 10:11:20.700953 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:40815
--local_ip_for_outbound_sockets=127.14.41.66
--tserver_master_addrs=127.14.41.126:32949
--webserver_port=40639
--webserver_interface=127.14.41.66
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:11:20.701520 30499 ts_tablet_manager.cc:1397] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.012s
Sep 25 10:11:20 dist-test-slave-s6tm krb5kdc[29839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331080, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:11:20.703224 30499 raft_consensus.cc:348] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } } peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } }
I20260925 10:11:20.703567 30499 raft_consensus.cc:729] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a9b1daf7c443a4acb86a081b74adde, State: Initialized, Role: FOLLOWER
I20260925 10:11:20.703717 30499 consensus_queue.cc:260] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } } peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } }
I20260925 10:11:20.704041 30499 ts_tablet_manager.cc:1428] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20260925 10:11:20.708844 30610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:20.694238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:56901 (local address 127.14.41.126:32949)
0925 10:11:20.696138 (+ 1900us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:20.696152 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:20.696201 (+ 49us) server_negotiation.cc:407] Connection header received
0925 10:11:20.696391 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:20.696402 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:20.696489 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:20.696678 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:11:20.697529 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.698929 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.700570 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.701227 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.702990 (+ 1763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:20.704269 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:11:20.704274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:11:20.704274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:11:20.704287 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:11:20.706492 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:20.707169 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:20.707173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:20.707173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:20.707290 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:20.707639 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:20.707640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:20.707641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:20.707990 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:11:20.708073 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:20.708499 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 10:11:20.708639 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1819,"thread_start_us":84,"threads_started":1}
I20260925 10:11:20.708920 30609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:20.694071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.65:56901)
0925 10:11:20.694288 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:20.694347 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:20.695153 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:20.696730 (+ 1577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:20.696736 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:20.697113 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:11:20.697380 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:20.697388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.699086 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.699088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:20.700441 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:20.700445 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.701341 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.701342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:20.701503 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:20.702187 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:11:20.702198 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:11:20.704075 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:11:20.706668 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:11:20.706678 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:11:20.706680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:11:20.707062 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:11:20.707375 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:11:20.707376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:11:20.707377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:11:20.707564 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:11:20.708068 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:11:20.708076 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:11:20.708293 (+ 217us) client_negotiation.cc:769] Sending connection context
0925 10:11:20.708428 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 10:11:20.708647 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":97,"threads_started":1}
I20260925 10:11:20.709710 30607 heartbeater.cc:347] Connected to a master server at 127.14.41.126:32949
I20260925 10:11:20.709836 30607 heartbeater.cc:464] Registering TS with master...
I20260925 10:11:20.710047 30607 heartbeater.cc:511] Master 127.14.41.126:32949 requested a full tablet report, sending...
I20260925 10:11:20.710927 30419 ts_manager.cc:194] Registered new tserver with Master: 19a9b1daf7c443a4acb86a081b74adde (127.14.41.65:43961)
I20260925 10:11:20.712164 30419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:56901
I20260925 10:11:20.713511 30607 heartbeater.cc:503] Master 127.14.41.126:32949 was elected leader, sending a full tablet report...
I20260925 10:11:20.719017 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:20.711597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:43961 (local address 127.0.0.1:51534)
0925 10:11:20.712758 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:20.712793 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:20.712863 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:20.716070 (+ 3207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:20.716074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:20.716076 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:11:20.716162 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:20.716164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.717040 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.717041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:20.717864 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:20.717867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.718659 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.718661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:20.718754 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:20.718772 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 10:11:20.718862 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 10:11:20.718936 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1131,"thread_start_us":118,"threads_started":1}
I20260925 10:11:20.719017 30615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:20.712557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51534 (local address 127.14.41.65:43961)
0925 10:11:20.715843 (+ 3286us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:20.715846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:20.715857 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:11:20.715895 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:20.715897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:20.715915 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:20.716049 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:11:20.716251 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.716944 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.718018 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.718564 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.718659 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:20.718722 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:20.718892 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 10:11:20.718934 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3253,"thread_start_us":101,"threads_started":1}
I20260925 10:11:20.719538 30541 tablet_service.cc:1514] Processing DeleteTablet for tablet d3213dccd4544ea5b56750b42fc98185 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:11:00 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:51534
I20260925 10:11:20.720325 30616 tablet_replica.cc:330] stopping tablet replica
I20260925 10:11:20.720546 30616 raft_consensus.cc:2229] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:11:20.720640 30616 raft_consensus.cc:2258] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:11:20.720994 30616 ts_tablet_manager.cc:1905] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:11:20.723272 30616 ts_tablet_manager.cc:1918] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:11:20.723369 30616 log.cc:1196] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/wal/wals/d3213dccd4544ea5b56750b42fc98185
I20260925 10:11:20.723744 30616 ts_tablet_manager.cc:1939] T d3213dccd4544ea5b56750b42fc98185 P 19a9b1daf7c443a4acb86a081b74adde: Deleting consensus metadata
I20260925 10:11:20.724359 30403 catalog_manager.cc:4841] TS 19a9b1daf7c443a4acb86a081b74adde (127.14.41.65:43961): tablet d3213dccd4544ea5b56750b42fc98185 (table test-table [id=67960c314ec24155bfa0c68e4694bc3e]) successfully deleted
W20260925 10:11:20.791308 30612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:11:20.791530 30612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:11:20.791555 30612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:11:20.792799 30612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:11:20.792853 30612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:11:20.792879 30612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:11:20.794322 30612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:40815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=40639
--tserver_master_addrs=127.14.41.126:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:11:20.794531 30612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:11:20.794713 30612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:11:20.800299 30622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:20.800371 30612 server_base.cc:1034] running on GCE node
W20260925 10:11:20.800434 30624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:20.800339 30621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:20.800691 30612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:11:20.800956 30612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:11:20.802193 30612 hybrid_clock.cc:648] HybridClock initialized: now 1790331080802144 us; error 67 us; skew 500 ppm
Sep 25 10:11:20 dist-test-slave-s6tm krb5kdc[29839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331080, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:11:20.805128 30612 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:11:20.805629 30612 webserver.cc:460] Webserver started at http://127.14.41.66:40639/ using document root <none> and password file <none>
I20260925 10:11:20.805894 30612 fs_manager.cc:359] Metadata directory not provided
I20260925 10:11:20.805992 30612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:11:20.807650 30612 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 10:11:20.808468 30631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:20.808671 30612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:11:20.808815 30612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "c9bbc45632c446d9a914a38f78c6c771"
format_stamp: "Formatted at 2026-09-25 10:11:00 on dist-test-slave-s6tm"
server_key: "e56118a112a827f1851c9944df779db7"
server_key_iv: "0296d7f8cb59eeca3c21a6e01e5d70ea"
server_key_version: "encryptionkey@0"
I20260925 10:11:20.809160 30612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:11:20.816659 30612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:11:20.819257 30612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:11:20.819418 30612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:11:20.819674 30612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:11:20.820072 30612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:11:20.820115 30612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:20.820138 30612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:11:20.820181 30612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:20.826190 30612 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:40815
I20260925 10:11:20.826287 30744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:40815 every 8 connection(s)
I20260925 10:11:20.826624 30612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:11:20.827804 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 30612
I20260925 10:11:20.828135 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:46465
--local_ip_for_outbound_sockets=127.14.41.67
--tserver_master_addrs=127.14.41.126:32949
--webserver_port=36917
--webserver_interface=127.14.41.67
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:11:20 dist-test-slave-s6tm krb5kdc[29839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331080, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:11:20.838135 30610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:20.827360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:34449 (local address 127.14.41.126:32949)
0925 10:11:20.827477 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:20.827480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:20.827655 (+ 175us) server_negotiation.cc:407] Connection header received
0925 10:11:20.828349 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:20.828351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:20.828367 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:20.828399 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:11:20.829620 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.830167 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.831814 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.832115 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.832182 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:20.834921 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:11:20.834922 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:11:20.834922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:11:20.834932 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:11:20.836602 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:20.837066 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:20.837068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:20.837068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:20.837135 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:20.837415 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:20.837416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:20.837417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:20.837536 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:11:20.837631 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:20.837932 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 10:11:20.838042 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 10:11:20.838335 30747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:20.827347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.66:34449)
0925 10:11:20.827505 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:20.827551 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:20.828202 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:20.828565 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:20.828568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:20.829163 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:11:20.829487 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:20.829494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.830273 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.830274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:20.831642 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:20.831645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.832226 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.832227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:20.832404 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:20.833034 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:11:20.833045 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:11:20.834807 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:11:20.836700 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:11:20.836709 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:11:20.836711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:11:20.836966 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:11:20.837206 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:11:20.837207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:11:20.837208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:11:20.837326 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:11:20.837625 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:11:20.837633 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:11:20.837782 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 10:11:20.837900 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 10:11:20.838067 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260925 10:11:20.838917 30745 heartbeater.cc:347] Connected to a master server at 127.14.41.126:32949
I20260925 10:11:20.839030 30745 heartbeater.cc:464] Registering TS with master...
I20260925 10:11:20.839231 30745 heartbeater.cc:511] Master 127.14.41.126:32949 requested a full tablet report, sending...
I20260925 10:11:20.839608 30419 ts_manager.cc:194] Registered new tserver with Master: c9bbc45632c446d9a914a38f78c6c771 (127.14.41.66:40815)
I20260925 10:11:20.840127 30419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:34449
W20260925 10:11:20.913717 30748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:11:20.913939 30748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:11:20.913964 30748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:11:20.915192 30748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:11:20.915239 30748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:11:20.915267 30748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:11:20.916785 30748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:46465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=36917
--tserver_master_addrs=127.14.41.126:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:11:20.917043 30748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:11:20.917295 30748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:11:20.922524 30754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:20.922549 30753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:20.922524 30756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:20.923174 30748 server_base.cc:1034] running on GCE node
I20260925 10:11:20.923355 30748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:11:20.923573 30748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:11:20.924714 30748 hybrid_clock.cc:648] HybridClock initialized: now 1790331080924703 us; error 32 us; skew 500 ppm
Sep 25 10:11:20 dist-test-slave-s6tm krb5kdc[29839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331080, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:11:20.927512 30748 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:11:20.928079 30748 webserver.cc:460] Webserver started at http://127.14.41.67:36917/ using document root <none> and password file <none>
I20260925 10:11:20.928324 30748 fs_manager.cc:359] Metadata directory not provided
I20260925 10:11:20.928392 30748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:11:20.930655 30748 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:11:20.931365 30763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:20.931581 30748 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:11:20.931689 30748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "e766569a581845a49536bca5db88c659"
format_stamp: "Formatted at 2026-09-25 10:11:00 on dist-test-slave-s6tm"
server_key: "9aca95e3863e4959ff33d24df752a428"
server_key_iv: "edb7cfdaed257442fb8b2ec35b5b8fae"
server_key_version: "encryptionkey@0"
I20260925 10:11:20.932129 30748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:11:20.953006 30748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:11:20.955592 30748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:11:20.955776 30748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:11:20.956096 30748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:11:20.956452 30748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:11:20.956509 30748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:20.956547 30748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:11:20.956573 30748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:20.962836 30748 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:46465
I20260925 10:11:20.962883 30876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:46465 every 8 connection(s)
I20260925 10:11:20.963245 30748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:11:20.964123 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 30748
Sep 25 10:11:20 dist-test-slave-s6tm krb5kdc[29839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331080, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:11:20.971870 30880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:20.964542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58304 (local address 127.14.41.126:32949)
0925 10:11:20.964671 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:20.964673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:20.964790 (+ 117us) server_negotiation.cc:407] Connection header received
0925 10:11:20.964878 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:20.964879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:20.964892 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:20.964972 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:11:20.965743 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.966427 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.967277 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.967482 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.967556 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:20.968913 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:11:20.968914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:11:20.968915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:11:20.968921 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:11:20.970515 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:20.970938 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:20.970939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:20.970939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:20.971030 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:20.971257 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:20.971258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:20.971258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:20.971414 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:11:20.971615 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:20.971668 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 10:11:20.971793 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260925 10:11:20.974186 30610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:20.964131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:36829 (local address 127.14.41.126:32949)
0925 10:11:20.964223 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:20.964226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:20.964475 (+ 249us) server_negotiation.cc:407] Connection header received
0925 10:11:20.965201 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:20.965202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:20.965209 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:20.965241 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:11:20.966377 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.966865 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.967891 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.968194 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.968266 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:20.970774 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:11:20.970776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:11:20.970776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:11:20.970784 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:11:20.972629 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:20.973182 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:20.973183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:20.973183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:20.973233 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:20.973580 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:20.973581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:20.973581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:20.973706 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:11:20.973773 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:20.974040 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 10:11:20.974119 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:11:20.974370 30879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:20.964168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.67:36829)
0925 10:11:20.964328 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:20.964385 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:20.965102 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:20.965341 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:20.965346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:20.966070 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:11:20.966271 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:20.966276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.966973 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.966974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:20.967770 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:20.967772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.968282 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.968283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:20.968403 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:20.968918 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:11:20.968927 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:11:20.970683 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:11:20.972760 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:11:20.972771 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:11:20.972773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:11:20.973115 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:11:20.973357 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:11:20.973357 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:11:20.973357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:11:20.973478 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:11:20.973763 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:11:20.973766 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:11:20.973892 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 10:11:20.973980 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 10:11:20.974116 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260925 10:11:20.975150 30877 heartbeater.cc:347] Connected to a master server at 127.14.41.126:32949
I20260925 10:11:20.975281 30877 heartbeater.cc:464] Registering TS with master...
I20260925 10:11:20.975561 30877 heartbeater.cc:511] Master 127.14.41.126:32949 requested a full tablet report, sending...
I20260925 10:11:20.976100 30419 ts_manager.cc:194] Registered new tserver with Master: e766569a581845a49536bca5db88c659 (127.14.41.67:46465)
I20260925 10:11:20.976557 30419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:36829
I20260925 10:11:20.984354 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:11:20.987777 30610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:20.984823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58312 (local address 127.14.41.126:32949)
0925 10:11:20.984900 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:20.984902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:20.985043 (+ 141us) server_negotiation.cc:407] Connection header received
0925 10:11:20.985088 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:20.985089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:20.985131 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:20.985203 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:11:20.985360 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.985851 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.986996 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.987186 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.987250 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:20.987459 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:11:20.987562 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:11:20.987629 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:20.987664 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 10:11:20.987681 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:11:20.988181 30419 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:58312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:11:20.989912 30419 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:11:20.992707 30541 tablet_service.cc:1467] Processing CreateTablet for tablet bc1e52173f2a4f03966112b82f367d4b (DEFAULT_TABLE table=test-table [id=d97406741ede447996a316ea431cfd93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:11:20.992946 30541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1e52173f2a4f03966112b82f367d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:11:20.994527 30499 tablet_bootstrap.cc:492] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde: Bootstrap starting.
I20260925 10:11:20.994867 30499 tablet_bootstrap.cc:654] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde: Neither blocks nor log segments found. Creating new log.
I20260925 10:11:20.995447 30499 tablet_bootstrap.cc:492] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde: No bootstrap required, opened a new log
I20260925 10:11:20.995514 30499 ts_tablet_manager.cc:1397] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:11:20.995644 30499 raft_consensus.cc:348] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } } peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } }
I20260925 10:11:20.995718 30499 raft_consensus.cc:374] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:11:20.995765 30499 raft_consensus.cc:729] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a9b1daf7c443a4acb86a081b74adde, State: Initialized, Role: FOLLOWER
I20260925 10:11:20.995818 30499 consensus_queue.cc:260] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } } peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } }
I20260925 10:11:20.995918 30499 ts_tablet_manager.cc:1428] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:11:20.996207 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:20.992029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:40815 (local address 127.0.0.1:36152)
0925 10:11:20.992062 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:20.992073 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:20.992110 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:20.992693 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:20.992694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:20.992695 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:11:20.992747 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:20.992749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.993705 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.993707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:20.994643 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:20.994645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.995706 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.995707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:20.995774 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:20.995779 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:11:20.995836 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 10:11:20.995896 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15,"mutex_wait_us":1}
I20260925 10:11:20.996239 30885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:20.992346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36152 (local address 127.14.41.66:40815)
0925 10:11:20.992540 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:20.992543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:20.992553 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:11:20.992597 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:20.992600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:20.992619 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:20.992699 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:11:20.992801 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.993610 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.994827 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.995622 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.995690 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:20.995750 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:20.995898 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 10:11:20.995977 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":108,"threads_started":1}
I20260925 10:11:20.996510 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:20.992034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:46465 (local address 127.0.0.1:42486)
0925 10:11:20.992511 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:20.992521 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:20.992582 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:20.992978 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:20.992979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:20.992982 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:11:20.993029 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:20.993031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.994206 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.994207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:20.995246 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:20.995249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.996260 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.996262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:20.996333 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:20.996337 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:11:20.996396 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 10:11:20.996440 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":125,"thread_start_us":131,"threads_started":1}
I20260925 10:11:20.996531 30884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:20.992131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42486 (local address 127.14.41.67:46465)
0925 10:11:20.992442 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:20.992447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:20.992580 (+ 133us) server_negotiation.cc:407] Connection header received
0925 10:11:20.992668 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:20.992671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:20.992687 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:20.992762 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:11:20.993117 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.994109 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.995343 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:20.996097 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:20.996168 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:20.996223 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:20.996414 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 10:11:20.996455 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":142,"threads_started":1}
I20260925 10:11:20.997253 30679 tablet_service.cc:1467] Processing CreateTablet for tablet bc1e52173f2a4f03966112b82f367d4b (DEFAULT_TABLE table=test-table [id=d97406741ede447996a316ea431cfd93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:11:20.997412 30811 tablet_service.cc:1467] Processing CreateTablet for tablet bc1e52173f2a4f03966112b82f367d4b (DEFAULT_TABLE table=test-table [id=d97406741ede447996a316ea431cfd93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:11:20.997673 30679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1e52173f2a4f03966112b82f367d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:11:20.997781 30811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc1e52173f2a4f03966112b82f367d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:11:21.000701 30887 tablet_bootstrap.cc:492] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659: Bootstrap starting.
I20260925 10:11:21.000936 30888 tablet_bootstrap.cc:492] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771: Bootstrap starting.
I20260925 10:11:21.001536 30888 tablet_bootstrap.cc:654] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771: Neither blocks nor log segments found. Creating new log.
I20260925 10:11:21.002063 30888 log.cc:824] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771: Log is configured to *not* fsync() on all Append() calls
I20260925 10:11:21.002964 30887 tablet_bootstrap.cc:654] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659: Neither blocks nor log segments found. Creating new log.
I20260925 10:11:21.002969 30888 tablet_bootstrap.cc:492] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771: No bootstrap required, opened a new log
I20260925 10:11:21.003041 30888 ts_tablet_manager.cc:1397] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:11:21.003584 30887 log.cc:824] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659: Log is configured to *not* fsync() on all Append() calls
I20260925 10:11:21.004482 30887 tablet_bootstrap.cc:492] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659: No bootstrap required, opened a new log
I20260925 10:11:21.004567 30887 ts_tablet_manager.cc:1397] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 10:11:21.005303 30888 raft_consensus.cc:348] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } } peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } }
I20260925 10:11:21.005504 30888 raft_consensus.cc:374] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:11:21.005543 30888 raft_consensus.cc:729] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9bbc45632c446d9a914a38f78c6c771, State: Initialized, Role: FOLLOWER
I20260925 10:11:21.005673 30888 consensus_queue.cc:260] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } } peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } }
I20260925 10:11:21.006027 30888 ts_tablet_manager.cc:1428] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:11:21.006081 30745 heartbeater.cc:503] Master 127.14.41.126:32949 was elected leader, sending a full tablet report...
I20260925 10:11:21.006714 30887 raft_consensus.cc:348] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } } peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } }
I20260925 10:11:21.006908 30887 raft_consensus.cc:374] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:11:21.006949 30887 raft_consensus.cc:729] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e766569a581845a49536bca5db88c659, State: Initialized, Role: FOLLOWER
I20260925 10:11:21.007076 30887 consensus_queue.cc:260] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } } peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } }
I20260925 10:11:21.007388 30877 heartbeater.cc:503] Master 127.14.41.126:32949 was elected leader, sending a full tablet report...
I20260925 10:11:21.007612 30887 ts_tablet_manager.cc:1428] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260925 10:11:21.077346 30746 tablet.cc:2369] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:11:21.161976 30892 raft_consensus.cc:482] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:11:21.162117 30892 raft_consensus.cc:504] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } } peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } }
I20260925 10:11:21.162498 30892 leader_election.cc:288] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9bbc45632c446d9a914a38f78c6c771 (127.14.41.66:40815), 19a9b1daf7c443a4acb86a081b74adde (127.14.41.65:43961)
I20260925 10:11:21.165668 30879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:21.162671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:43961 (local address 127.14.41.67:51729)
0925 10:11:21.162730 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:21.162743 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:21.162790 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:21.163029 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:21.163030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:21.163032 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:11:21.163107 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:21.163109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:21.163835 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:21.163836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:21.164810 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:21.164813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:21.165383 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:21.165384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:21.165466 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:21.165469 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:11:21.165546 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 10:11:21.165589 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":2}
I20260925 10:11:21.165668 30615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:21.162768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:51729 (local address 127.14.41.65:43961)
0925 10:11:21.162809 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:21.162811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:21.162820 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:11:21.162859 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:21.162860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:21.162912 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:21.162995 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:11:21.163188 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:21.163736 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:21.164912 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:21.165280 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:21.165383 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:21.165392 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:21.165555 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 10:11:21.165595 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:11:21.165786 30885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:21.162752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:33917 (local address 127.14.41.66:40815)
0925 10:11:21.162804 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:21.162806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:21.162969 (+ 163us) server_negotiation.cc:407] Connection header received
0925 10:11:21.163082 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:21.163083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:21.163092 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:21.163160 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:11:21.163320 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:21.163839 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:21.164926 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:21.165306 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:21.165444 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:21.165453 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:21.165640 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 10:11:21.165688 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 10:11:21.166031 30893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:21.162700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:40815 (local address 127.14.41.67:33917)
0925 10:11:21.162881 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:21.162891 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:21.163014 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:21.163162 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:21.163163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:21.163165 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:11:21.163236 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:21.163237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:21.163987 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:21.163988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:21.164812 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:21.164814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:21.165444 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:21.165445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:21.165524 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:21.165530 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:11:21.165893 (+ 363us) client_negotiation.cc:240] Negotiation successful
0925 10:11:21.165948 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":5,"thread_start_us":95,"threads_started":1}
I20260925 10:11:21.166023 30561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1e52173f2a4f03966112b82f367d4b" candidate_uuid: "e766569a581845a49536bca5db88c659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a9b1daf7c443a4acb86a081b74adde" is_pre_election: true
I20260925 10:11:21.166200 30561 raft_consensus.cc:2454] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e766569a581845a49536bca5db88c659 in term 0.
I20260925 10:11:21.166441 30764 leader_election.cc:302] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19a9b1daf7c443a4acb86a081b74adde, e766569a581845a49536bca5db88c659; no voters:
I20260925 10:11:21.166457 30699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1e52173f2a4f03966112b82f367d4b" candidate_uuid: "e766569a581845a49536bca5db88c659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9bbc45632c446d9a914a38f78c6c771" is_pre_election: true
I20260925 10:11:21.166596 30699 raft_consensus.cc:2454] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e766569a581845a49536bca5db88c659 in term 0.
I20260925 10:11:21.166596 30892 raft_consensus.cc:2781] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:11:21.166642 30892 raft_consensus.cc:482] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:11:21.166666 30892 raft_consensus.cc:3037] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:11:21.167443 30892 raft_consensus.cc:504] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } } peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } }
I20260925 10:11:21.167609 30892 leader_election.cc:288] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [CANDIDATE]: Term 1 election: Requested vote from peers c9bbc45632c446d9a914a38f78c6c771 (127.14.41.66:40815), 19a9b1daf7c443a4acb86a081b74adde (127.14.41.65:43961)
I20260925 10:11:21.167727 30699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1e52173f2a4f03966112b82f367d4b" candidate_uuid: "e766569a581845a49536bca5db88c659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9bbc45632c446d9a914a38f78c6c771"
I20260925 10:11:21.167806 30699 raft_consensus.cc:3037] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:11:21.167766 30561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc1e52173f2a4f03966112b82f367d4b" candidate_uuid: "e766569a581845a49536bca5db88c659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a9b1daf7c443a4acb86a081b74adde"
I20260925 10:11:21.167966 30561 raft_consensus.cc:3037] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:11:21.168586 30699 raft_consensus.cc:2454] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e766569a581845a49536bca5db88c659 in term 1.
I20260925 10:11:21.168780 30767 leader_election.cc:302] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9bbc45632c446d9a914a38f78c6c771, e766569a581845a49536bca5db88c659; no voters:
I20260925 10:11:21.168859 30892 raft_consensus.cc:2781] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:11:21.169030 30892 raft_consensus.cc:686] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [term 1 LEADER]: Becoming Leader. State: Replica: e766569a581845a49536bca5db88c659, State: Running, Role: LEADER
I20260925 10:11:21.169134 30892 consensus_queue.cc:237] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } } peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } }
I20260925 10:11:21.169188 30561 raft_consensus.cc:2454] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e766569a581845a49536bca5db88c659 in term 1.
I20260925 10:11:21.170130 30419 catalog_manager.cc:5495] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 reported cstate change: term changed from 0 to 1, leader changed from <none> to e766569a581845a49536bca5db88c659 (127.14.41.67). New cstate: current_term: 1 leader_uuid: "e766569a581845a49536bca5db88c659" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } health_report { overall_health: UNKNOWN } } }
I20260925 10:11:21.181068 30884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:21.178525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42500 (local address 127.14.41.67:46465)
0925 10:11:21.178593 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:21.178595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:21.178604 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:11:21.178645 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:21.178646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:21.178663 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:21.178742 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:11:21.178878 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:21.179358 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:21.180242 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:21.180447 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:21.180533 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:21.180714 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:11:21.180769 (+ 55us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 10:11:21.180947 (+ 178us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42500: token expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 10:11:21.181134 30884 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42500: token expired
I20260925 10:11:21.187881 30610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:21.181477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58318 (local address 127.14.41.126:32949)
0925 10:11:21.181528 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:21.181531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:21.181556 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:11:21.181578 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:21.181579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:21.181596 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:21.181650 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:11:21.182176 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:21.182646 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:21.183413 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:21.183607 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:21.183692 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:21.185004 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:11:21.185006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:11:21.185006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:11:21.185015 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:11:21.186565 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:21.187010 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:21.187011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:21.187012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:21.187080 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:21.187285 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:21.187285 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:21.187286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:21.187446 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:11:21.187532 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:21.187681 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 10:11:21.187809 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:11:21.191524 30884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:21.189118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42514 (local address 127.14.41.67:46465)
0925 10:11:21.189205 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:21.189213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:21.189223 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:11:21.189249 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:21.189250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:21.189263 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:21.189333 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:11:21.189496 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:21.189973 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:21.190748 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:21.190935 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:21.191006 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:21.191167 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:11:21.191271 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:11:21.191319 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:21.191422 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 10:11:21.191445 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:11:21.193648 30561 raft_consensus.cc:1261] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Refusing update from remote peer e766569a581845a49536bca5db88c659: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:11:21.193682 30699 raft_consensus.cc:1261] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771 [term 1 FOLLOWER]: Refusing update from remote peer e766569a581845a49536bca5db88c659: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:11:21.194095 30894 consensus_queue.cc:1035] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:11:21.194293 30892 consensus_queue.cc:1035] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 10:11:21.195642 30608 tablet.cc:2369] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:11:21.196940 30897 mvcc.cc:205] Tried to move back new op lower bound from 7333196108566167552 to 7333196108469633024. Current Snapshot: MvccSnapshot[applied={T|T < 7333196108566167552}]
I20260925 10:11:21.198405 30896 mvcc.cc:205] Tried to move back new op lower bound from 7333196108566167552 to 7333196108469633024. Current Snapshot: MvccSnapshot[applied={T|T < 7333196108566167552 or (T in {7333196108566167552})}]
I20260925 10:11:21.199581 30418 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:11:21.199676 30418 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:11:21.200512 30418 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17: Sending DeleteTablet for 3 replicas of tablet bc1e52173f2a4f03966112b82f367d4b
I20260925 10:11:21.200796 30811 tablet_service.cc:1514] Processing DeleteTablet for tablet bc1e52173f2a4f03966112b82f367d4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:11:21 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:42486
I20260925 10:11:21.200860 30541 tablet_service.cc:1514] Processing DeleteTablet for tablet bc1e52173f2a4f03966112b82f367d4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:11:21 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:51534
I20260925 10:11:21.200959 30616 tablet_replica.cc:330] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde: stopping tablet replica
I20260925 10:11:21.201005 30616 raft_consensus.cc:2229] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:11:21.201050 30616 raft_consensus.cc:2258] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:11:21.201040 30679 tablet_service.cc:1514] Processing DeleteTablet for tablet bc1e52173f2a4f03966112b82f367d4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:11:21 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:36152
I20260925 10:11:21.201284 30906 tablet_replica.cc:330] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771: stopping tablet replica
I20260925 10:11:21.201349 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 30457
I20260925 10:11:21.201399 30906 raft_consensus.cc:2229] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:11:21.201654 30616 ts_tablet_manager.cc:1905] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:11:21.201923 30906 raft_consensus.cc:2258] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:11:21.202335 30906 ts_tablet_manager.cc:1905] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:11:21.204106 30906 ts_tablet_manager.cc:1918] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:11:21.204305 30906 log.cc:1196] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/bc1e52173f2a4f03966112b82f367d4b
I20260925 10:11:21.204635 30906 ts_tablet_manager.cc:1939] T bc1e52173f2a4f03966112b82f367d4b P c9bbc45632c446d9a914a38f78c6c771: Deleting consensus metadata
I20260925 10:11:21.205173 30406 catalog_manager.cc:4841] TS c9bbc45632c446d9a914a38f78c6c771 (127.14.41.66:40815): tablet bc1e52173f2a4f03966112b82f367d4b (table test-table [id=d97406741ede447996a316ea431cfd93]) successfully deleted
I20260925 10:11:21.208141 30905 tablet_replica.cc:330] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659: stopping tablet replica
I20260925 10:11:21.208410 30905 raft_consensus.cc:2229] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:11:21.208563 30905 raft_consensus.cc:2258] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:11:21.209046 30905 ts_tablet_manager.cc:1905] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 10:11:21.209893 30403 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.41.65:43961 (error 108)
I20260925 10:11:21.210278 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 30612
I20260925 10:11:21.210865 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:21.210312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:43961 (local address 127.0.0.1:51538)
0925 10:11:21.210766 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:21.210820 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.65:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423}
W20260925 10:11:21.211105 30403 catalog_manager.cc:4584] TS 19a9b1daf7c443a4acb86a081b74adde (127.14.41.65:43961): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc1e52173f2a4f03966112b82f367d4b: Network error: Client connection negotiation failed: client connection to 127.14.41.65:43961: connect: Connection refused (error 111)
I20260925 10:11:21.211830 30905 ts_tablet_manager.cc:1918] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:11:21.211930 30905 log.cc:1196] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/bc1e52173f2a4f03966112b82f367d4b
I20260925 10:11:21.212263 30905 ts_tablet_manager.cc:1939] T bc1e52173f2a4f03966112b82f367d4b P e766569a581845a49536bca5db88c659: Deleting consensus metadata
I20260925 10:11:21.212946 30403 catalog_manager.cc:4841] TS e766569a581845a49536bca5db88c659 (127.14.41.67:46465): tablet bc1e52173f2a4f03966112b82f367d4b (table test-table [id=d97406741ede447996a316ea431cfd93]) successfully deleted
I20260925 10:11:21.216598 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 30748
I20260925 10:11:21.222261 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 30386
I20260925 10:11:41.228168 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:32949
--webserver_interface=127.14.41.126
--webserver_port=39877
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:32949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:11:41.316414 30908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:11:41.316634 30908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:11:41.316658 30908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:11:41.317811 30908 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 10:11:41.317847 30908 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 10:11:41.317860 30908 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 10:11:41.317873 30908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:11:41.317885 30908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:11:41.317898 30908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:11:41.317910 30908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:11:41.317922 30908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:11:41.319345 30908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:32949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:32949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:11:41.319543 30908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:11:41.319717 30908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:11:41.324905 30913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:41.324976 30916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:41.325098 30908 server_base.cc:1034] running on GCE node
W20260925 10:11:41.324935 30914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:41.325388 30908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:11:41.325664 30908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:11:41.326810 30908 hybrid_clock.cc:648] HybridClock initialized: now 1790331101326792 us; error 41 us; skew 500 ppm
Sep 25 10:11:41 dist-test-slave-s6tm krb5kdc[29839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331101, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:11:41.333251 30908 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:11:41.333799 30908 webserver.cc:460] Webserver started at http://127.14.41.126:39877/ using document root <none> and password file <none>
I20260925 10:11:41.334111 30908 fs_manager.cc:359] Metadata directory not provided
I20260925 10:11:41.334184 30908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:11:41.335659 30908 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:11:41.336583 30923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:41.336766 30908 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:11:41.336875 30908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "a6ca323bab744bcd93cef175968c3c17"
format_stamp: "Formatted at 2026-09-25 10:10:59 on dist-test-slave-s6tm"
server_key: "1c58dfb79a831334b621f512fd1771d6"
server_key_iv: "0eafa4e6a160e55f9c60b1a3f663f773"
server_key_version: "encryptionkey@0"
I20260925 10:11:41.337253 30908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:11:41.348207 30908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:11:41.350819 30908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:11:41.350991 30908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:11:41.356029 30908 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:32949
I20260925 10:11:41.356069 30975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:32949 every 8 connection(s)
I20260925 10:11:41.356467 30908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:11:41.358035 30976 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:11:41.358845 30976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17: Bootstrap starting.
I20260925 10:11:41.364568 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 30908
I20260925 10:11:41.364948 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:43961
--local_ip_for_outbound_sockets=127.14.41.65
--tserver_master_addrs=127.14.41.126:32949
--webserver_port=40233
--webserver_interface=127.14.41.65
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:11:41.368637 30976 log.cc:824] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17: Log is configured to *not* fsync() on all Append() calls
I20260925 10:11:41.371196 30976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:11:41.371575 30976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17: Bootstrap complete.
I20260925 10:11:41.373978 30976 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ca323bab744bcd93cef175968c3c17" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 32949 } }
I20260925 10:11:41.374260 30976 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6ca323bab744bcd93cef175968c3c17, State: Initialized, Role: FOLLOWER
I20260925 10:11:41.374491 30976 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ca323bab744bcd93cef175968c3c17" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 32949 } }
I20260925 10:11:41.374593 30976 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:11:41.374650 30976 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:11:41.374699 30976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 2 FOLLOWER]: Advancing to term 3
I20260925 10:11:41.375797 30976 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ca323bab744bcd93cef175968c3c17" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 32949 } }
I20260925 10:11:41.375924 30976 leader_election.cc:302] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6ca323bab744bcd93cef175968c3c17; no voters:
I20260925 10:11:41.376158 30976 leader_election.cc:288] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 10:11:41.376199 30980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 10:11:41.376315 30980 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [term 3 LEADER]: Becoming Leader. State: Replica: a6ca323bab744bcd93cef175968c3c17, State: Running, Role: LEADER
I20260925 10:11:41.376415 30980 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ca323bab744bcd93cef175968c3c17" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 32949 } }
I20260925 10:11:41.376469 30976 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:11:41.376715 30981 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a6ca323bab744bcd93cef175968c3c17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ca323bab744bcd93cef175968c3c17" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 32949 } } }
I20260925 10:11:41.376811 30981 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [sys.catalog]: This master's current role is: LEADER
I20260925 10:11:41.377069 30982 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6ca323bab744bcd93cef175968c3c17. Latest consensus state: current_term: 3 leader_uuid: "a6ca323bab744bcd93cef175968c3c17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ca323bab744bcd93cef175968c3c17" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 32949 } } }
I20260925 10:11:41.377173 30982 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17 [sys.catalog]: This master's current role is: LEADER
I20260925 10:11:41.377312 30985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:11:41.378159 30985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:11:41.378273 30985 catalog_manager.cc:1259] Loaded cluster ID: 74d160872e6d4e1f891407efe9bd4293
I20260925 10:11:41.378306 30985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:11:41.379559 30985 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:11:41.379765 30985 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17: Loaded TSK: 0
I20260925 10:11:41.379812 30985 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a6ca323bab744bcd93cef175968c3c17: Loaded TSK: 1
F20260925 10:11:41.380136 30985 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc5b12ff9ad google::LogMessage::Fail() at ??:0
@ 0x7fc5b130288a google::LogMessage::SendToLog() at ??:0
@ 0x7fc5b12ff4de google::LogMessage::Flush() at ??:0
@ 0x7fc5b1300a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc5b3a6c104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc5b3a69d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fc5b92591cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fc5b925948e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fc5b9259594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fc5b921fc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fc5b9229ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc5b31cd0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc5b31c2954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc5b40266db start_thread at ??:0
@ 0x7fc5afdfd61f clone at ??:0
W20260925 10:11:41.474511 30978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:11:41.474740 30978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:11:41.474766 30978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:11:41.476130 30978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:11:41.476183 30978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:11:41.476214 30978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:11:41.477816 30978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:43961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=40233
--tserver_master_addrs=127.14.41.126:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:11:41.478129 30978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:11:41.478392 30978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:11:41.483897 31018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:41.483894 31017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:41.484124 30978 server_base.cc:1034] running on GCE node
W20260925 10:11:41.483919 31020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:41.484668 30978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:11:41.484951 30978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:11:41.486128 30978 hybrid_clock.cc:648] HybridClock initialized: now 1790331101486114 us; error 32 us; skew 500 ppm
Sep 25 10:11:41 dist-test-slave-s6tm krb5kdc[29839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331101, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:11:41.488937 30978 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:11:41.489472 30978 webserver.cc:460] Webserver started at http://127.14.41.65:40233/ using document root <none> and password file <none>
I20260925 10:11:41.489714 30978 fs_manager.cc:359] Metadata directory not provided
I20260925 10:11:41.489789 30978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:11:41.491307 30978 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:11:41.492226 31027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:41.492435 30978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:11:41.492558 30978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "19a9b1daf7c443a4acb86a081b74adde"
format_stamp: "Formatted at 2026-09-25 10:10:59 on dist-test-slave-s6tm"
server_key: "6ebde7883f7cedacee793c1e7e115cf6"
server_key_iv: "0f8bb9acfd7df3d2dd0e353e20c32653"
server_key_version: "encryptionkey@0"
I20260925 10:11:41.492992 30978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:11:41.508955 30978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:11:41.511286 30978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:11:41.511469 30978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:11:41.511726 30978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:11:41.512328 31034 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 10:11:41.514385 30978 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 10:11:41.514463 30978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 10:11:41.514528 30978 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 10:11:41.515254 30978 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 10:11:41.515313 30978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 10:11:41.515399 31034 tablet_bootstrap.cc:492] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde: Bootstrap starting.
I20260925 10:11:41.517366 31034 log.cc:824] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde: Log is configured to *not* fsync() on all Append() calls
I20260925 10:11:41.520397 31034 tablet_bootstrap.cc:492] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:11:41.520817 31034 tablet_bootstrap.cc:492] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde: Bootstrap complete.
I20260925 10:11:41.520982 31034 ts_tablet_manager.cc:1397] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260925 10:11:41.522842 30978 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:43961
I20260925 10:11:41.522830 31034 raft_consensus.cc:348] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } } peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } }
I20260925 10:11:41.523180 31034 raft_consensus.cc:729] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a9b1daf7c443a4acb86a081b74adde, State: Initialized, Role: FOLLOWER
I20260925 10:11:41.523296 30978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:11:41.523370 31034 consensus_queue.cc:260] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } } peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } }
I20260925 10:11:41.523681 31034 ts_tablet_manager.cc:1428] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260925 10:11:41.523900 31142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:43961 every 8 connection(s)
I20260925 10:11:41.530927 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 30978
I20260925 10:11:41.531286 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:40815
--local_ip_for_outbound_sockets=127.14.41.66
--tserver_master_addrs=127.14.41.126:32949
--webserver_port=40639
--webserver_interface=127.14.41.66
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:11:41.593868 31146 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:11:41.524555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.65:51347)
0925 10:11:41.524720 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:41.524754 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:41.524921 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:41.593691 (+ 68770us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
W20260925 10:11:41.594141 31143 heartbeater.cc:644] Failed to heartbeat to 127.14.41.126:32949 (0 consecutive failures): Network error: Failed to ping master at 127.14.41.126:32949: Client connection negotiation failed: client connection to 127.14.41.126:32949: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 10:11:41.594498 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:41.594386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.65:56543)
0925 10:11:41.594442 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:41.594481 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 10:11:41.594847 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:41.594757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.65:59399)
0925 10:11:41.594810 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:41.594832 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 10:11:41.594954 31143 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:11:41.628069 31147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:11:41.628312 31147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:11:41.628355 31147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:11:41.629591 31147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:11:41.629665 31147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:11:41.629727 31147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:11:41.631254 31147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:40815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=40639
--tserver_master_addrs=127.14.41.126:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:11:41.631522 31147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:11:41.631759 31147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:11:41.637156 31155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:41.637202 31153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:41.637271 31147 server_base.cc:1034] running on GCE node
W20260925 10:11:41.637234 31152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:41.637562 31147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:11:41.637854 31147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:11:41.639031 31147 hybrid_clock.cc:648] HybridClock initialized: now 1790331101639028 us; error 39 us; skew 500 ppm
Sep 25 10:11:41 dist-test-slave-s6tm krb5kdc[29839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331101, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:11:41.641709 31147 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:11:41.642174 31147 webserver.cc:460] Webserver started at http://127.14.41.66:40639/ using document root <none> and password file <none>
I20260925 10:11:41.642442 31147 fs_manager.cc:359] Metadata directory not provided
I20260925 10:11:41.642514 31147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:11:41.643857 31147 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:11:41.644505 31162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:41.644698 31147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:11:41.644807 31147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "c9bbc45632c446d9a914a38f78c6c771"
format_stamp: "Formatted at 2026-09-25 10:11:00 on dist-test-slave-s6tm"
server_key: "e56118a112a827f1851c9944df779db7"
server_key_iv: "0296d7f8cb59eeca3c21a6e01e5d70ea"
server_key_version: "encryptionkey@0"
I20260925 10:11:41.645232 31147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:11:41.661981 31147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:11:41.664371 31147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:11:41.664556 31147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:11:41.664834 31147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:11:41.665176 31147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:11:41.665239 31147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:41.665287 31147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:11:41.665311 31147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:41.671435 31147 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:40815
I20260925 10:11:41.671516 31275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:40815 every 8 connection(s)
I20260925 10:11:41.671892 31147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:11:41.673070 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:41.672671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.66:59239)
0925 10:11:41.672843 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:41.672924 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":65,"threads_started":1}
W20260925 10:11:41.673555 31276 heartbeater.cc:644] Failed to heartbeat to 127.14.41.126:32949 (0 consecutive failures): Network error: Failed to ping master at 127.14.41.126:32949: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
I20260925 10:11:41.673863 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:41.673766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.66:38785)
0925 10:11:41.673815 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:41.673844 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 10:11:41.674129 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:41.674058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.66:46557)
0925 10:11:41.674095 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:41.674115 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 10:11:41.674487 31276 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:11:41.677652 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 31147
I20260925 10:11:41.677920 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:46465
--local_ip_for_outbound_sockets=127.14.41.67
--tserver_master_addrs=127.14.41.126:32949
--webserver_port=36917
--webserver_interface=127.14.41.67
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:11:41.765714 31279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:11:41.765930 31279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:11:41.765954 31279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:11:41.767059 31279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:11:41.767096 31279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:11:41.767122 31279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:11:41.768651 31279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:46465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=36917
--tserver_master_addrs=127.14.41.126:32949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:11:41.768869 31279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:11:41.769053 31279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:11:41.774327 31284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:41.774420 31285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:41.774364 31287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:41.774598 31279 server_base.cc:1034] running on GCE node
I20260925 10:11:41.774816 31279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:11:41.775086 31279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:11:41.776301 31279 hybrid_clock.cc:648] HybridClock initialized: now 1790331101776288 us; error 46 us; skew 500 ppm
Sep 25 10:11:41 dist-test-slave-s6tm krb5kdc[29839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331101, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:11:41.779014 31279 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:11:41.779486 31279 webserver.cc:460] Webserver started at http://127.14.41.67:36917/ using document root <none> and password file <none>
I20260925 10:11:41.779747 31279 fs_manager.cc:359] Metadata directory not provided
I20260925 10:11:41.779824 31279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:11:41.781399 31279 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:11:41.782008 31294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:41.782192 31279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 10:11:41.782372 31279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "e766569a581845a49536bca5db88c659"
format_stamp: "Formatted at 2026-09-25 10:11:00 on dist-test-slave-s6tm"
server_key: "9aca95e3863e4959ff33d24df752a428"
server_key_iv: "edb7cfdaed257442fb8b2ec35b5b8fae"
server_key_version: "encryptionkey@0"
I20260925 10:11:41.782745 31279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:11:41.799701 31279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:11:41.801919 31279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:11:41.802105 31279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:11:41.802374 31279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:11:41.802709 31279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:11:41.802766 31279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:41.802800 31279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:11:41.802829 31279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:41.809322 31279 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:46465
I20260925 10:11:41.809381 31407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:46465 every 8 connection(s)
I20260925 10:11:41.809749 31279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:11:41.810874 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:41.810428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.67:40003)
0925 10:11:41.810620 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:41.810712 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":75,"threads_started":1}
W20260925 10:11:41.811374 31408 heartbeater.cc:644] Failed to heartbeat to 127.14.41.126:32949 (0 consecutive failures): Network error: Failed to ping master at 127.14.41.126:32949: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
I20260925 10:11:41.811676 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:41.811582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.67:44011)
0925 10:11:41.811627 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:41.811656 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 10:11:41.812029 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:41.811909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.67:41555)
0925 10:11:41.811988 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:41.812015 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 10:11:41.812166 31408 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:11:41.813208 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 31279
I20260925 10:11:42.595530 31412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:42.595275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.65:40115)
0925 10:11:42.595433 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:42.595477 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260925 10:11:42.675177 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:42.674929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.66:33767)
0925 10:11:42.675066 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:42.675097 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260925 10:11:42.812839 31414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:42.812561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.67:34203)
0925 10:11:42.812699 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:42.812782 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260925 10:11:43.155194 31415 raft_consensus.cc:482] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:11:43.155341 31415 raft_consensus.cc:504] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } } peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } }
I20260925 10:11:43.155630 31415 leader_election.cc:288] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e766569a581845a49536bca5db88c659 (127.14.41.67:46465), c9bbc45632c446d9a914a38f78c6c771 (127.14.41.66:40815)
Sep 25 10:11:43 dist-test-slave-s6tm krb5kdc[29839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331101, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.67@KRBTEST.COM
Sep 25 10:11:43 dist-test-slave-s6tm krb5kdc[29839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331101, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.66@KRBTEST.COM
I20260925 10:11:43.167634 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:43.155967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:51827 (local address 127.14.41.67:46465)
0925 10:11:43.156211 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:43.156223 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:43.156261 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:11:43.156686 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:43.156688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:43.156726 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:43.156885 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:11:43.157965 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:43.158860 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:43.159850 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:43.160340 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:43.160475 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:43.163415 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:11:43.163419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:11:43.163419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:11:43.163432 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:11:43.165798 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:43.166204 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:43.166208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:43.166208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:43.166328 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:43.166606 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:43.166607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:43.166608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:43.166960 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:11:43.167078 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:43.167389 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 10:11:43.167516 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":61,"threads_started":1}
I20260925 10:11:43.167634 31416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:43.155827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:46465 (local address 127.14.41.65:51827)
0925 10:11:43.155981 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:43.155993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:43.156058 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:43.156912 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:43.156915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:43.157329 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:11:43.157835 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:43.157841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:43.158971 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:43.158973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:43.159744 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:43.159746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:43.160436 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:43.160437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:43.160570 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:43.161274 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:11:43.161287 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:11:43.163221 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:11:43.165876 (+ 2655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:11:43.165884 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:11:43.165886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:11:43.166113 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:11:43.166429 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:11:43.166429 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:11:43.166430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:11:43.166533 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:11:43.167058 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:11:43.167066 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:11:43.167208 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 10:11:43.167319 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 10:11:43.167488 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":2,"thread_start_us":60,"threads_started":1}
I20260925 10:11:43.168478 31417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:43.155827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:40815 (local address 127.14.41.65:57225)
0925 10:11:43.156010 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:43.156021 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:43.156090 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:43.156913 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:43.156916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:43.157314 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:11:43.159728 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:43.159732 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:43.160972 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:43.160973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:43.161846 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:43.161850 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:43.162703 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:43.162704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:43.162793 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:43.163268 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:11:43.163273 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:11:43.164699 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:11:43.166934 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:11:43.166934 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:11:43.166935 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:11:43.167119 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:11:43.167522 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:11:43.167523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:11:43.167524 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:11:43.167604 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:11:43.168115 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:11:43.168115 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:11:43.168160 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 10:11:43.168254 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 10:11:43.168355 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":40,"thread_start_us":56,"threads_started":1}
I20260925 10:11:43.168587 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:43.155997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:57225 (local address 127.14.41.66:40815)
0925 10:11:43.156156 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:43.156169 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:43.156215 (+ 46us) server_negotiation.cc:407] Connection header received
0925 10:11:43.156679 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:43.156681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:43.156726 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:43.156885 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:11:43.159871 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:43.160868 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:43.161941 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:43.162590 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:43.162714 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:43.164804 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:11:43.164808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:11:43.164808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:11:43.164818 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:11:43.166829 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:43.167340 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:43.167343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:43.167344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:43.167439 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:43.167679 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:43.167680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:43.167681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:43.168054 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:11:43.168134 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:43.168315 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 10:11:43.168451 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
W20260925 10:11:43.168744 31028 leader_election.cc:341] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e766569a581845a49536bca5db88c659 (127.14.41.67:46465): Not found: Tablet not found: bc1e52173f2a4f03966112b82f367d4b
W20260925 10:11:43.169363 31031 leader_election.cc:341] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c9bbc45632c446d9a914a38f78c6c771 (127.14.41.66:40815): Not found: Tablet not found: bc1e52173f2a4f03966112b82f367d4b
I20260925 10:11:43.169428 31031 leader_election.cc:302] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19a9b1daf7c443a4acb86a081b74adde; no voters: c9bbc45632c446d9a914a38f78c6c771, e766569a581845a49536bca5db88c659
I20260925 10:11:43.169529 31415 raft_consensus.cc:2726] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:11:43.596364 31417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:43.596204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.65:55255)
0925 10:11:43.596278 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:43.596320 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 10:11:43.675832 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:43.675632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.66:56655)
0925 10:11:43.675754 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:43.675786 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260925 10:11:43.813630 31421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:43.813402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.67:43391)
0925 10:11:43.813547 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:43.813583 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260925 10:11:44.597254 31422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:44.597027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.65:33847)
0925 10:11:44.597159 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:44.597202 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260925 10:11:44.676735 31423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:44.676473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.66:34599)
0925 10:11:44.676632 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:44.676684 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260925 10:11:44.814460 31424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:44.814173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.67:60993)
0925 10:11:44.814356 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:44.814412 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260925 10:11:45.305963 31425 raft_consensus.cc:482] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:11:45.306092 31425 raft_consensus.cc:504] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } } peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } }
I20260925 10:11:45.306412 31425 leader_election.cc:288] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e766569a581845a49536bca5db88c659 (127.14.41.67:46465), c9bbc45632c446d9a914a38f78c6c771 (127.14.41.66:40815)
W20260925 10:11:45.306980 31031 leader_election.cc:341] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c9bbc45632c446d9a914a38f78c6c771 (127.14.41.66:40815): Not found: Tablet not found: bc1e52173f2a4f03966112b82f367d4b
W20260925 10:11:45.307116 31028 leader_election.cc:341] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e766569a581845a49536bca5db88c659 (127.14.41.67:46465): Not found: Tablet not found: bc1e52173f2a4f03966112b82f367d4b
I20260925 10:11:45.307230 31028 leader_election.cc:302] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19a9b1daf7c443a4acb86a081b74adde; no voters: c9bbc45632c446d9a914a38f78c6c771, e766569a581845a49536bca5db88c659
I20260925 10:11:45.307336 31425 raft_consensus.cc:2726] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:11:45.598079 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:45.597791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.65:48997)
0925 10:11:45.597969 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:45.598016 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":76,"threads_started":1}
I20260925 10:11:45.677891 31427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:45.677533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.66:53619)
0925 10:11:45.677765 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:45.677817 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":96,"threads_started":1}
I20260925 10:11:45.815282 31428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:45.815024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.67:55805)
0925 10:11:45.815175 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:45.815214 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":64,"threads_started":1}
I20260925 10:11:46.599159 31429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:46.598820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.65:59659)
0925 10:11:46.599062 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:46.599109 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":149,"threads_started":1}
I20260925 10:11:46.678938 31430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:46.678651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.66:44745)
0925 10:11:46.678837 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:46.678884 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
I20260925 10:11:46.816294 31431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:46.816014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.67:49349)
0925 10:11:46.816165 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:46.816242 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":61,"threads_started":1}
I20260925 10:11:47.361848 31432 raft_consensus.cc:482] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:11:47.361984 31432 raft_consensus.cc:504] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } } peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } }
I20260925 10:11:47.362210 31432 leader_election.cc:288] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e766569a581845a49536bca5db88c659 (127.14.41.67:46465), c9bbc45632c446d9a914a38f78c6c771 (127.14.41.66:40815)
W20260925 10:11:47.362584 31028 leader_election.cc:341] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e766569a581845a49536bca5db88c659 (127.14.41.67:46465): Not found: Tablet not found: bc1e52173f2a4f03966112b82f367d4b
W20260925 10:11:47.362660 31031 leader_election.cc:341] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c9bbc45632c446d9a914a38f78c6c771 (127.14.41.66:40815): Not found: Tablet not found: bc1e52173f2a4f03966112b82f367d4b
I20260925 10:11:47.362744 31031 leader_election.cc:302] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19a9b1daf7c443a4acb86a081b74adde; no voters: c9bbc45632c446d9a914a38f78c6c771, e766569a581845a49536bca5db88c659
I20260925 10:11:47.362852 31432 raft_consensus.cc:2726] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:11:47.600160 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:47.599795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.65:46597)
0925 10:11:47.600043 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:47.600095 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":67,"threads_started":1}
I20260925 10:11:47.680055 31434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:47.679715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.66:54717)
0925 10:11:47.679952 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:47.679998 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":97,"threads_started":1}
I20260925 10:11:47.817159 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:47.816909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.67:43123)
0925 10:11:47.817062 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:47.817109 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260925 10:11:48.601114 31436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:48.600836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.65:40969)
0925 10:11:48.601006 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:48.601058 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":75,"threads_started":1}
I20260925 10:11:48.680951 31437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:48.680696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.66:50603)
0925 10:11:48.680846 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:48.680891 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
I20260925 10:11:48.818071 31438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:48.817805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.67:48127)
0925 10:11:48.817972 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:48.818022 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
I20260925 10:11:49.602152 31439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:49.601871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.65:54525)
0925 10:11:49.602046 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:49.602102 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":75,"threads_started":1}
I20260925 10:11:49.682000 31440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:49.681709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.66:48619)
0925 10:11:49.681899 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:49.681948 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":99,"threads_started":1}
I20260925 10:11:49.819080 31441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:49.818841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.67:46847)
0925 10:11:49.818986 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:49.819032 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260925 10:11:50.603111 31442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:50.602845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.65:48015)
0925 10:11:50.603014 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:50.603061 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":66,"threads_started":1}
I20260925 10:11:50.682962 31443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:50.682609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.66:45089)
0925 10:11:50.682826 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:50.682893 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":68,"threads_started":1}
I20260925 10:11:50.820011 31444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:50.819653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.67:48417)
0925 10:11:50.819794 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:50.819892 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
I20260925 10:11:51.489863 31445 raft_consensus.cc:482] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:11:51.490020 31445 raft_consensus.cc:504] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } } peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } }
I20260925 10:11:51.490262 31445 leader_election.cc:288] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e766569a581845a49536bca5db88c659 (127.14.41.67:46465), c9bbc45632c446d9a914a38f78c6c771 (127.14.41.66:40815)
W20260925 10:11:51.490720 31028 leader_election.cc:341] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e766569a581845a49536bca5db88c659 (127.14.41.67:46465): Not found: Tablet not found: bc1e52173f2a4f03966112b82f367d4b
W20260925 10:11:51.490804 31031 leader_election.cc:341] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c9bbc45632c446d9a914a38f78c6c771 (127.14.41.66:40815): Not found: Tablet not found: bc1e52173f2a4f03966112b82f367d4b
I20260925 10:11:51.490948 31031 leader_election.cc:302] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19a9b1daf7c443a4acb86a081b74adde; no voters: c9bbc45632c446d9a914a38f78c6c771, e766569a581845a49536bca5db88c659
I20260925 10:11:51.491056 31445 raft_consensus.cc:2726] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:11:51.604159 31446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:51.603750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.65:47003)
0925 10:11:51.604048 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:51.604105 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
I20260925 10:11:51.684266 31447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:51.683704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.66:35097)
0925 10:11:51.684044 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:51.684207 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":95,"threads_started":1}
I20260925 10:11:51.821148 31448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:51.820857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.67:47515)
0925 10:11:51.821039 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:51.821078 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":66,"threads_started":1}
I20260925 10:11:52.605080 31449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:52.604804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.65:51449)
0925 10:11:52.604981 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:52.605029 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":66,"threads_started":1}
I20260925 10:11:52.685390 31450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:52.685020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.66:39777)
0925 10:11:52.685169 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:52.685215 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260925 10:11:52.822033 31451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:52.821767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.67:57591)
0925 10:11:52.821941 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:52.821982 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":71,"threads_started":1}
I20260925 10:11:53.117017 31452 raft_consensus.cc:482] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:11:53.117123 31452 raft_consensus.cc:504] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e766569a581845a49536bca5db88c659" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 46465 } } peers { permanent_uuid: "c9bbc45632c446d9a914a38f78c6c771" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 40815 } } peers { permanent_uuid: "19a9b1daf7c443a4acb86a081b74adde" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 43961 } }
I20260925 10:11:53.117359 31452 leader_election.cc:288] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e766569a581845a49536bca5db88c659 (127.14.41.67:46465), c9bbc45632c446d9a914a38f78c6c771 (127.14.41.66:40815)
W20260925 10:11:53.117923 31031 leader_election.cc:341] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c9bbc45632c446d9a914a38f78c6c771 (127.14.41.66:40815): Not found: Tablet not found: bc1e52173f2a4f03966112b82f367d4b
W20260925 10:11:53.117998 31028 leader_election.cc:341] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e766569a581845a49536bca5db88c659 (127.14.41.67:46465): Not found: Tablet not found: bc1e52173f2a4f03966112b82f367d4b
I20260925 10:11:53.118072 31028 leader_election.cc:302] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19a9b1daf7c443a4acb86a081b74adde; no voters: c9bbc45632c446d9a914a38f78c6c771, e766569a581845a49536bca5db88c659
I20260925 10:11:53.118166 31452 raft_consensus.cc:2726] T bc1e52173f2a4f03966112b82f367d4b P 19a9b1daf7c443a4acb86a081b74adde [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:11:53.606026 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:53.605779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.65:57079)
0925 10:11:53.605924 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:53.605974 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":69,"threads_started":1}
I20260925 10:11:53.686285 31454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:53.685960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.66:47911)
0925 10:11:53.686185 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:53.686234 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":72,"threads_started":1}
I20260925 10:11:53.822885 31455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:53.822644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.67:48727)
0925 10:11:53.822782 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:53.822828 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260925 10:11:54.606824 31456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:54.606592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.65:44107)
0925 10:11:54.606728 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:54.606775 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260925 10:11:54.687299 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:54.687012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.66:34031)
0925 10:11:54.687161 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:54.687210 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":69,"threads_started":1}
I20260925 10:11:54.823681 31458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:54.823408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.67:56633)
0925 10:11:54.823588 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:54.823632 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":77,"threads_started":1}
I20260925 10:11:55.607600 31459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:55.607382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.65:38807)
0925 10:11:55.607506 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:55.607549 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260925 10:11:55.688115 31460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:55.687834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.66:55771)
0925 10:11:55.688015 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:55.688064 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260925 10:11:55.824568 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:55.824230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.67:38639)
0925 10:11:55.824439 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:55.824514 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":113,"threads_started":1}
I20260925 10:11:56.608634 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:56.608385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.65:41061)
0925 10:11:56.608536 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:56.608584 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260925 10:11:56.688921 31463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:56.688699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:32949 (local address 127.14.41.66:47561)
0925 10:11:56.688835 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:56.688872 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.126:32949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260925 10:11:56.815056 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 30978
I20260925 10:11:56.820016 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 31147
I20260925 10:11:56.824645 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 31279
2026-09-25T10:11:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58409 ms)
[----------] 30 tests from SecurityITest (239351 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:11:56 dist-test-slave-s6tm krb5kdc[31466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:11:56 dist-test-slave-s6tm krb5kdc[31466](info): set up 2 sockets
Sep 25 10:11:56 dist-test-slave-s6tm krb5kdc[31466](info): commencing operation
krb5kdc: starting...
W20260925 10:11:57.873404 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:11:57 dist-test-slave-s6tm krb5kdc[31466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:11:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:11:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:11:58.000272 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:44229
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:42073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:44229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:11:58.087972 31479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:11:58.088264 31479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:11:58.088315 31479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:11:58.089581 31479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:11:58.089663 31479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:11:58.089689 31479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:11:58.089720 31479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:11:58.089746 31479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:11:58.091331 31479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:42073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:44229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:44229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:11:58.091622 31479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:11:58.091881 31479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:11:58.097725 31487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:58.097719 31484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:58.097729 31479 server_base.cc:1034] running on GCE node
W20260925 10:11:58.097715 31485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:58.098420 31479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:11:58.098712 31479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:11:58.099865 31479 hybrid_clock.cc:648] HybridClock initialized: now 1790331118099854 us; error 36 us; skew 500 ppm
Sep 25 10:11:58 dist-test-slave-s6tm krb5kdc[31466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331118, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:11:58.102663 31479 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:11:58.103206 31479 webserver.cc:460] Webserver started at http://127.14.41.126:37311/ using document root <none> and password file <none>
I20260925 10:11:58.103477 31479 fs_manager.cc:359] Metadata directory not provided
I20260925 10:11:58.103551 31479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:11:58.103703 31479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:11:58.104887 31479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "04711e99e0734b569e6d7ffd9d33713f"
format_stamp: "Formatted at 2026-09-25 10:11:58 on dist-test-slave-s6tm"
server_key: "256e34ccc286b91d529b4a0f2bc0f0fd"
server_key_iv: "d0742b81f563e29bbc4a7687f5347d37"
server_key_version: "encryptionkey@0"
I20260925 10:11:58.105266 31479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "04711e99e0734b569e6d7ffd9d33713f"
format_stamp: "Formatted at 2026-09-25 10:11:58 on dist-test-slave-s6tm"
server_key: "256e34ccc286b91d529b4a0f2bc0f0fd"
server_key_iv: "d0742b81f563e29bbc4a7687f5347d37"
server_key_version: "encryptionkey@0"
I20260925 10:11:58.106639 31479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:11:58.107455 31494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:58.107635 31479 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:11:58.107745 31479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "04711e99e0734b569e6d7ffd9d33713f"
format_stamp: "Formatted at 2026-09-25 10:11:58 on dist-test-slave-s6tm"
server_key: "256e34ccc286b91d529b4a0f2bc0f0fd"
server_key_iv: "d0742b81f563e29bbc4a7687f5347d37"
server_key_version: "encryptionkey@0"
I20260925 10:11:58.107821 31479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:11:58.125172 31479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:11:58.127734 31479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:11:58.127918 31479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:11:58.132306 31479 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:44229
I20260925 10:11:58.132359 31546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:44229 every 8 connection(s)
I20260925 10:11:58.132750 31479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:11:58.133632 31547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:11:58.135710 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 31479
I20260925 10:11:58.135819 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:11:58.136050 14501 external_mini_cluster.cc:1428] Setting key 0f441ee6e8ac933778b1602501eadad7
I20260925 10:11:58.136413 31547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f: Bootstrap starting.
I20260925 10:11:58.137137 31547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f: Neither blocks nor log segments found. Creating new log.
I20260925 10:11:58.137436 31547 log.cc:824] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:11:58.138154 31547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f: No bootstrap required, opened a new log
I20260925 10:11:58.140089 31547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04711e99e0734b569e6d7ffd9d33713f" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44229 } }
I20260925 10:11:58.140259 31547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:11:58.140308 31547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04711e99e0734b569e6d7ffd9d33713f, State: Initialized, Role: FOLLOWER
I20260925 10:11:58.140450 31547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04711e99e0734b569e6d7ffd9d33713f" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44229 } }
I20260925 10:11:58.140535 31547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:11:58.140579 31547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:11:58.140628 31547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:11:58.141182 31547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04711e99e0734b569e6d7ffd9d33713f" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44229 } }
I20260925 10:11:58.141314 31547 leader_election.cc:302] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04711e99e0734b569e6d7ffd9d33713f; no voters:
I20260925 10:11:58.141507 31547 leader_election.cc:288] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:11:58.141530 31552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:11:58.141645 31552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f [term 1 LEADER]: Becoming Leader. State: Replica: 04711e99e0734b569e6d7ffd9d33713f, State: Running, Role: LEADER
I20260925 10:11:58.141748 31552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04711e99e0734b569e6d7ffd9d33713f" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44229 } }
I20260925 10:11:58.141799 31547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:11:58.142098 31552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04711e99e0734b569e6d7ffd9d33713f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04711e99e0734b569e6d7ffd9d33713f" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44229 } } }
I20260925 10:11:58.142315 31552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f [sys.catalog]: This master's current role is: LEADER
I20260925 10:11:58.142513 31552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04711e99e0734b569e6d7ffd9d33713f. Latest consensus state: current_term: 1 leader_uuid: "04711e99e0734b569e6d7ffd9d33713f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04711e99e0734b569e6d7ffd9d33713f" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44229 } } }
I20260925 10:11:58.142633 31552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f [sys.catalog]: This master's current role is: LEADER
I20260925 10:11:58.142974 31561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 10:11:58 dist-test-slave-s6tm krb5kdc[31466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:11:58.143625 31561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:11:58.146637 31561 catalog_manager.cc:1347] Generated new cluster ID: 9956cc09a5904ab683df9034d9c0f5de
I20260925 10:11:58.146723 31561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:11:58.147818 31550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.136607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.14.41.126:44229)
0925 10:11:58.136787 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:58.136797 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:58.136830 (+ 33us) server_negotiation.cc:407] Connection header received
0925 10:11:58.137348 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:58.137358 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:58.137427 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:58.137580 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:11:58.138316 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.139255 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.140120 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.140726 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.140877 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:58.143867 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:11:58.143871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:11:58.143871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:11:58.143883 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:11:58.146104 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:58.146506 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:58.146509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:58.146509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:58.146648 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:58.146870 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:58.146871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:58.146872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:58.147168 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:11:58.147328 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:58.147472 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 10:11:58.147603 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260925 10:11:58.160315 31561 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:11:58.161010 31561 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:11:58.166424 31561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f: Generated new TSK 0
I20260925 10:11:58.166646 31561 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
I20260925 10:11:58.198988 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:44229
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:42073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:11:58.303922 31573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:11:58.304211 31573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:11:58.304236 31573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:11:58.305459 31573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:11:58.305506 31573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:11:58.305533 31573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:11:58.307040 31573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:42073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:44229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:11:58.307294 31573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:11:58.307557 31573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:11:58.313501 31579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:58.313508 31581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:58.313508 31578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:58.313573 31573 server_base.cc:1034] running on GCE node
I20260925 10:11:58.313884 31573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:11:58.314149 31573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:11:58.315313 31573 hybrid_clock.cc:648] HybridClock initialized: now 1790331118315304 us; error 33 us; skew 500 ppm
Sep 25 10:11:58 dist-test-slave-s6tm krb5kdc[31466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331118, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:11:58.317907 31573 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:11:58.318384 31573 webserver.cc:460] Webserver started at http://127.14.41.65:44303/ using document root <none> and password file <none>
I20260925 10:11:58.318641 31573 fs_manager.cc:359] Metadata directory not provided
I20260925 10:11:58.318714 31573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:11:58.318876 31573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:11:58.320014 31573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "d2b31ded5b2c485ea0914cab92a28565"
format_stamp: "Formatted at 2026-09-25 10:11:58 on dist-test-slave-s6tm"
server_key: "baf9d7035a325731f1b65d72c026a5bc"
server_key_iv: "e111c583568545899981d48260c3418e"
server_key_version: "encryptionkey@0"
I20260925 10:11:58.320395 31573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "d2b31ded5b2c485ea0914cab92a28565"
format_stamp: "Formatted at 2026-09-25 10:11:58 on dist-test-slave-s6tm"
server_key: "baf9d7035a325731f1b65d72c026a5bc"
server_key_iv: "e111c583568545899981d48260c3418e"
server_key_version: "encryptionkey@0"
I20260925 10:11:58.321746 31573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:11:58.322458 31588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:58.322636 31573 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 10:11:58.322741 31573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "d2b31ded5b2c485ea0914cab92a28565"
format_stamp: "Formatted at 2026-09-25 10:11:58 on dist-test-slave-s6tm"
server_key: "baf9d7035a325731f1b65d72c026a5bc"
server_key_iv: "e111c583568545899981d48260c3418e"
server_key_version: "encryptionkey@0"
I20260925 10:11:58.322816 31573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:11:58.349436 31573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:11:58.351528 31573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:11:58.351703 31573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:11:58.351986 31573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:11:58.352344 31573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:11:58.352401 31573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:58.352437 31573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:11:58.352485 31573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:58.358166 31573 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:37859
I20260925 10:11:58.358245 31701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:37859 every 8 connection(s)
I20260925 10:11:58.358611 31573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
Sep 25 10:11:58 dist-test-slave-s6tm krb5kdc[31466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331118, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:11:58.365458 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 31573
I20260925 10:11:58.365571 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:11:58.365792 14501 external_mini_cluster.cc:1428] Setting key 90d3fd2970187d1bdb9c7758ea0c8f96
I20260925 10:11:58.369328 31550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.359424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:45957 (local address 127.14.41.126:44229)
0925 10:11:58.359512 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:58.359514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:58.359685 (+ 171us) server_negotiation.cc:407] Connection header received
0925 10:11:58.360356 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:58.360358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:58.360370 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:58.360422 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:11:58.361488 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.362034 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.363087 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.363319 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.363403 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:58.365865 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:11:58.365867 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:11:58.365867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:11:58.365877 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:11:58.367725 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:58.368241 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:58.368242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:58.368243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:58.368316 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:58.368595 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:58.368596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:58.368596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:58.368714 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:11:58.368831 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:58.369130 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 10:11:58.369221 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 10:11:58.369576 31704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.359333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:44229 (local address 127.14.41.65:45957)
0925 10:11:58.359552 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:58.359598 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:58.360233 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:58.360509 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:58.360513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:58.361117 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:11:58.361375 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:58.361381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.362149 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.362155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:58.362987 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:58.362990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.363413 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.363415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:58.363547 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:58.364156 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:11:58.364164 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:11:58.365762 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:11:58.367831 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:11:58.367840 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:11:58.367842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:11:58.368153 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:11:58.368393 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:11:58.368394 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:11:58.368394 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:11:58.368516 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:11:58.368838 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:11:58.368846 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:11:58.368991 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 10:11:58.369099 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:11:58.369284 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":84,"threads_started":1}
I20260925 10:11:58.370041 31702 heartbeater.cc:347] Connected to a master server at 127.14.41.126:44229
I20260925 10:11:58.370137 31702 heartbeater.cc:464] Registering TS with master...
I20260925 10:11:58.370344 31702 heartbeater.cc:511] Master 127.14.41.126:44229 requested a full tablet report, sending...
I20260925 10:11:58.370921 31511 ts_manager.cc:194] Registered new tserver with Master: d2b31ded5b2c485ea0914cab92a28565 (127.14.41.65:37859)
I20260925 10:11:58.371543 31511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:45957
WARNING: no policy specified for kudu/127.14.41.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
I20260925 10:11:58.394887 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:44229
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:42073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:11:58.479239 31707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:11:58.479518 31707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:11:58.479560 31707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:11:58.480903 31707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:11:58.480991 31707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:11:58.481037 31707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:11:58.482645 31707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:42073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:44229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:11:58.482959 31707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:11:58.483204 31707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:11:58.488726 31712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:58.488754 31713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:58.489025 31707 server_base.cc:1034] running on GCE node
W20260925 10:11:58.488770 31715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:58.489332 31707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:11:58.489552 31707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:11:58.490679 31707 hybrid_clock.cc:648] HybridClock initialized: now 1790331118490668 us; error 28 us; skew 500 ppm
Sep 25 10:11:58 dist-test-slave-s6tm krb5kdc[31466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331118, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:11:58.493278 31707 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:11:58.493788 31707 webserver.cc:460] Webserver started at http://127.14.41.66:36683/ using document root <none> and password file <none>
I20260925 10:11:58.493986 31707 fs_manager.cc:359] Metadata directory not provided
I20260925 10:11:58.494027 31707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:11:58.494120 31707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:11:58.495153 31707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "8be712cca7974a2497a341045de442ba"
format_stamp: "Formatted at 2026-09-25 10:11:58 on dist-test-slave-s6tm"
server_key: "00a609d5a2033c39bc55619728bb2ce0"
server_key_iv: "6cceb87e41df55b96f685777f6dca211"
server_key_version: "encryptionkey@0"
I20260925 10:11:58.495546 31707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "8be712cca7974a2497a341045de442ba"
format_stamp: "Formatted at 2026-09-25 10:11:58 on dist-test-slave-s6tm"
server_key: "00a609d5a2033c39bc55619728bb2ce0"
server_key_iv: "6cceb87e41df55b96f685777f6dca211"
server_key_version: "encryptionkey@0"
I20260925 10:11:58.496970 31707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:11:58.497694 31722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:58.497907 31707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:11:58.498013 31707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "8be712cca7974a2497a341045de442ba"
format_stamp: "Formatted at 2026-09-25 10:11:58 on dist-test-slave-s6tm"
server_key: "00a609d5a2033c39bc55619728bb2ce0"
server_key_iv: "6cceb87e41df55b96f685777f6dca211"
server_key_version: "encryptionkey@0"
I20260925 10:11:58.498090 31707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:11:58.510931 31707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:11:58.513339 31707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:11:58.513515 31707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:11:58.513765 31707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:11:58.514086 31707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:11:58.514139 31707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:58.514181 31707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:11:58.514204 31707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:58.520668 31707 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:38499
I20260925 10:11:58.520713 31835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:38499 every 8 connection(s)
I20260925 10:11:58.521047 31707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:11:58 dist-test-slave-s6tm krb5kdc[31466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331118, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:11:58.530563 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 31707
I20260925 10:11:58.530710 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:11:58.530879 31550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.521782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:34233 (local address 127.14.41.126:44229)
0925 10:11:58.521856 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:58.521859 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:58.522048 (+ 189us) server_negotiation.cc:407] Connection header received
0925 10:11:58.522655 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:58.522656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:58.522668 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:58.522723 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:11:58.523720 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.524289 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.525297 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.525518 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.525617 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:58.527881 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:11:58.527882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:11:58.527882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:11:58.527890 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:11:58.529267 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:58.529713 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:58.529714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:58.529715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:58.529778 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:58.530041 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:58.530042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:58.530042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:58.530165 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:11:58.530240 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:58.530631 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 10:11:58.530801 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:11:58.530964 14501 external_mini_cluster.cc:1428] Setting key 2a8c23ff88291613967f4bbd029106ca
I20260925 10:11:58.531106 31838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.521738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:44229 (local address 127.14.41.66:34233)
0925 10:11:58.521895 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:58.521945 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:58.522535 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:58.522813 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:58.522822 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:58.523412 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:11:58.523613 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:58.523618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.524389 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.524390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:58.525201 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:58.525204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.525637 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.525639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:58.525771 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:58.526337 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:11:58.526346 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:11:58.527772 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:11:58.529397 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:11:58.529404 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:11:58.529406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:11:58.529618 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:11:58.529856 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:11:58.529857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:11:58.529857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:11:58.529969 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:11:58.530242 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:11:58.530250 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:11:58.530440 (+ 190us) client_negotiation.cc:769] Sending connection context
0925 10:11:58.530573 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 10:11:58.530776 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260925 10:11:58.531612 31836 heartbeater.cc:347] Connected to a master server at 127.14.41.126:44229
I20260925 10:11:58.531733 31836 heartbeater.cc:464] Registering TS with master...
I20260925 10:11:58.531960 31836 heartbeater.cc:511] Master 127.14.41.126:44229 requested a full tablet report, sending...
I20260925 10:11:58.532954 31511 ts_manager.cc:194] Registered new tserver with Master: 8be712cca7974a2497a341045de442ba (127.14.41.66:38499)
I20260925 10:11:58.533465 31511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:34233
WARNING: no policy specified for kudu/127.14.41.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
I20260925 10:11:58.559412 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:44229
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:42073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:11:58.645552 31841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:11:58.645763 31841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:11:58.645787 31841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:11:58.646978 31841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:11:58.647020 31841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:11:58.647073 31841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:11:58.648586 31841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:42073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:44229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:11:58.648851 31841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:11:58.649078 31841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:11:58.654548 31846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:58.654536 31847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:58.654539 31849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:58.654575 31841 server_base.cc:1034] running on GCE node
I20260925 10:11:58.655146 31841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:11:58.655419 31841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:11:58.656603 31841 hybrid_clock.cc:648] HybridClock initialized: now 1790331118656585 us; error 32 us; skew 500 ppm
Sep 25 10:11:58 dist-test-slave-s6tm krb5kdc[31466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331118, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:11:58.659294 31841 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:11:58.659740 31841 webserver.cc:460] Webserver started at http://127.14.41.67:41411/ using document root <none> and password file <none>
I20260925 10:11:58.660061 31841 fs_manager.cc:359] Metadata directory not provided
I20260925 10:11:58.660132 31841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:11:58.660291 31841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:11:58.661383 31841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "413dcea634094f0bbd95f5d6a08c9f6f"
format_stamp: "Formatted at 2026-09-25 10:11:58 on dist-test-slave-s6tm"
server_key: "1e88f653620b337e45bb2d7f05e518d2"
server_key_iv: "07d0a53ddaba8f85d9cd50474c94a77b"
server_key_version: "encryptionkey@0"
I20260925 10:11:58.661757 31841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "413dcea634094f0bbd95f5d6a08c9f6f"
format_stamp: "Formatted at 2026-09-25 10:11:58 on dist-test-slave-s6tm"
server_key: "1e88f653620b337e45bb2d7f05e518d2"
server_key_iv: "07d0a53ddaba8f85d9cd50474c94a77b"
server_key_version: "encryptionkey@0"
I20260925 10:11:58.663100 31841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:11:58.663980 31856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:58.664167 31841 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:11:58.664288 31841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "413dcea634094f0bbd95f5d6a08c9f6f"
format_stamp: "Formatted at 2026-09-25 10:11:58 on dist-test-slave-s6tm"
server_key: "1e88f653620b337e45bb2d7f05e518d2"
server_key_iv: "07d0a53ddaba8f85d9cd50474c94a77b"
server_key_version: "encryptionkey@0"
I20260925 10:11:58.664377 31841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:11:58.672623 31841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:11:58.674757 31841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:11:58.674928 31841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:11:58.675297 31841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:11:58.675650 31841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:11:58.675712 31841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:58.675756 31841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:11:58.675783 31841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:58.681737 31841 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:34691
I20260925 10:11:58.681782 31969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:34691 every 8 connection(s)
I20260925 10:11:58.682143 31841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:11:58.684655 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 31841
I20260925 10:11:58.684784 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:11:58.684974 14501 external_mini_cluster.cc:1428] Setting key 34a2dc79482119546f9107552fcf32f8
Sep 25 10:11:58 dist-test-slave-s6tm krb5kdc[31466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331118, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:11:58.691581 31550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.682905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:51343 (local address 127.14.41.126:44229)
0925 10:11:58.682983 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:58.682985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:58.683199 (+ 214us) server_negotiation.cc:407] Connection header received
0925 10:11:58.683671 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:58.683672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:58.683682 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:58.683749 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:11:58.684849 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.685422 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.686420 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.686612 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.686670 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:58.688935 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:11:58.688938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:11:58.688938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:11:58.688944 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:11:58.690351 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:58.690741 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:58.690741 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:58.690742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:58.690800 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:58.691031 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:58.691032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:58.691032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:58.691140 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:11:58.691193 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:58.691446 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 10:11:58.691510 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:11:58.691788 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.682887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:44229 (local address 127.14.41.67:51343)
0925 10:11:58.683055 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:58.683097 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:58.683583 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:58.683794 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:58.683797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:58.684474 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:11:58.684713 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:58.684721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.685518 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.685518 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:58.686333 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:58.686335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.686737 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.686738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:58.686850 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:58.687332 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:11:58.687341 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:11:58.688844 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:11:58.690429 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:11:58.690436 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:11:58.690438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:11:58.690669 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:11:58.690866 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:11:58.690867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:11:58.690867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:11:58.690972 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:11:58.691198 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:11:58.691203 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:11:58.691328 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 10:11:58.691415 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 10:11:58.691539 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260925 10:11:58.692584 31970 heartbeater.cc:347] Connected to a master server at 127.14.41.126:44229
I20260925 10:11:58.692678 31970 heartbeater.cc:464] Registering TS with master...
I20260925 10:11:58.692842 31970 heartbeater.cc:511] Master 127.14.41.126:44229 requested a full tablet report, sending...
I20260925 10:11:58.693218 31511 ts_manager.cc:194] Registered new tserver with Master: 413dcea634094f0bbd95f5d6a08c9f6f (127.14.41.67:34691)
I20260925 10:11:58.693557 31511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:51343
I20260925 10:11:58.696666 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:11:58.704200 31550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.697995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38842 (local address 127.14.41.126:44229)
0925 10:11:58.698061 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:58.698063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:58.698168 (+ 105us) server_negotiation.cc:407] Connection header received
0925 10:11:58.698210 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:58.698211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:58.698220 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:58.698262 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:11:58.698822 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.699330 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.700146 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.700355 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.700409 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:58.701694 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:11:58.701695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:11:58.701695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:11:58.701701 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:11:58.703047 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:58.703369 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:58.703370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:58.703370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:58.703421 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:11:58.703611 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:11:58.703611 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:11:58.703612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:11:58.703752 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:11:58.703830 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:58.704014 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 10:11:58.704077 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:11:58.705945 31511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:11:58.707577 31511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:11:58.714826 31986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.710491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54582 (local address 127.14.41.66:38499)
0925 10:11:58.710657 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:58.710663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:58.710995 (+ 332us) server_negotiation.cc:407] Connection header received
0925 10:11:58.711054 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:58.711057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:58.711072 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:58.711144 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:11:58.711414 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.712521 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.713901 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.714404 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.714500 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:58.714565 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:58.714697 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 10:11:58.714737 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260925 10:11:58.714819 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.710181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:38499 (local address 127.0.0.1:54582)
0925 10:11:58.710879 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:58.710888 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:58.710931 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:58.711267 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:58.711268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:58.711270 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:11:58.711315 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:58.711316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.712623 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.712624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:58.713800 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:58.713803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.714501 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.714502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:58.714581 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:58.714596 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 10:11:58.714669 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 10:11:58.714728 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":30,"thread_start_us":102,"threads_started":1}
I20260925 10:11:58.714941 31984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.710278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35640 (local address 127.14.41.67:34691)
0925 10:11:58.710482 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:58.710488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:58.710745 (+ 257us) server_negotiation.cc:407] Connection header received
0925 10:11:58.710866 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:58.710869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:58.710889 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:58.710966 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:11:58.711254 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.712488 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.713703 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.714478 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.714554 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:58.714613 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:58.714791 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 10:11:58.714836 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
I20260925 10:11:58.715394 31985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.710461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53138 (local address 127.14.41.65:37859)
0925 10:11:58.710606 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:58.710610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:58.710761 (+ 151us) server_negotiation.cc:407] Connection header received
0925 10:11:58.710867 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:58.710870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:58.710888 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:58.711263 (+ 375us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:11:58.711333 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.712158 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.713890 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.714611 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.714679 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:58.714728 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:58.715262 (+ 534us) server_negotiation.cc:299] Negotiation successful
0925 10:11:58.715309 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260925 10:11:58.715289 31983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.710167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:34691 (local address 127.0.0.1:35640)
0925 10:11:58.710677 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:58.710687 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:58.710766 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:58.711065 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:58.711068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:58.711071 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:11:58.711164 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:58.711167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.712618 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.712619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:58.713609 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:58.713612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.714652 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.714653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:58.714716 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:58.714721 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:11:58.714775 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 10:11:58.714830 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":45,"thread_start_us":61,"threads_started":1}
I20260925 10:11:58.715607 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.710167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:37859 (local address 127.0.0.1:53138)
0925 10:11:58.710633 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:58.710675 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:58.710771 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:58.711088 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:58.711089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:58.711091 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:11:58.711165 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:58.711169 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.712281 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.712282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:58.713806 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:58.713807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.715093 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.715093 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:58.715183 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:58.715187 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:11:58.715250 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 10:11:58.715291 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":89,"threads_started":1}
I20260925 10:11:58.716060 31770 tablet_service.cc:1467] Processing CreateTablet for tablet 5c9d7c186a804f19b8f9f1235f33e66e (DEFAULT_TABLE table=test-table [id=779ff0eb384b451eb34d949df0b44c1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:11:58.716456 31770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c9d7c186a804f19b8f9f1235f33e66e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:11:58.716408 31904 tablet_service.cc:1467] Processing CreateTablet for tablet 5c9d7c186a804f19b8f9f1235f33e66e (DEFAULT_TABLE table=test-table [id=779ff0eb384b451eb34d949df0b44c1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:11:58.716435 31636 tablet_service.cc:1467] Processing CreateTablet for tablet 5c9d7c186a804f19b8f9f1235f33e66e (DEFAULT_TABLE table=test-table [id=779ff0eb384b451eb34d949df0b44c1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:11:58.716729 31904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c9d7c186a804f19b8f9f1235f33e66e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:11:58.716760 31636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c9d7c186a804f19b8f9f1235f33e66e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:11:58.719487 31987 tablet_bootstrap.cc:492] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f: Bootstrap starting.
I20260925 10:11:58.720466 31988 tablet_bootstrap.cc:492] T 5c9d7c186a804f19b8f9f1235f33e66e P d2b31ded5b2c485ea0914cab92a28565: Bootstrap starting.
I20260925 10:11:58.720579 31987 tablet_bootstrap.cc:654] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f: Neither blocks nor log segments found. Creating new log.
I20260925 10:11:58.721194 31988 tablet_bootstrap.cc:654] T 5c9d7c186a804f19b8f9f1235f33e66e P d2b31ded5b2c485ea0914cab92a28565: Neither blocks nor log segments found. Creating new log.
I20260925 10:11:58.721202 31987 log.cc:824] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:11:58.721861 31988 log.cc:824] T 5c9d7c186a804f19b8f9f1235f33e66e P d2b31ded5b2c485ea0914cab92a28565: Log is configured to *not* fsync() on all Append() calls
I20260925 10:11:58.722167 31987 tablet_bootstrap.cc:492] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f: No bootstrap required, opened a new log
I20260925 10:11:58.722200 31989 tablet_bootstrap.cc:492] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba: Bootstrap starting.
I20260925 10:11:58.722250 31987 ts_tablet_manager.cc:1397] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:11:58.722712 31988 tablet_bootstrap.cc:492] T 5c9d7c186a804f19b8f9f1235f33e66e P d2b31ded5b2c485ea0914cab92a28565: No bootstrap required, opened a new log
I20260925 10:11:58.722839 31988 ts_tablet_manager.cc:1397] T 5c9d7c186a804f19b8f9f1235f33e66e P d2b31ded5b2c485ea0914cab92a28565: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:11:58.722896 31989 tablet_bootstrap.cc:654] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba: Neither blocks nor log segments found. Creating new log.
I20260925 10:11:58.723479 31989 log.cc:824] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba: Log is configured to *not* fsync() on all Append() calls
I20260925 10:11:58.723697 31987 raft_consensus.cc:348] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413dcea634094f0bbd95f5d6a08c9f6f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34691 } } peers { permanent_uuid: "8be712cca7974a2497a341045de442ba" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38499 } } peers { permanent_uuid: "d2b31ded5b2c485ea0914cab92a28565" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 37859 } }
I20260925 10:11:58.723966 31987 raft_consensus.cc:374] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:11:58.724020 31987 raft_consensus.cc:729] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413dcea634094f0bbd95f5d6a08c9f6f, State: Initialized, Role: FOLLOWER
I20260925 10:11:58.724177 31987 consensus_queue.cc:260] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413dcea634094f0bbd95f5d6a08c9f6f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34691 } } peers { permanent_uuid: "8be712cca7974a2497a341045de442ba" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38499 } } peers { permanent_uuid: "d2b31ded5b2c485ea0914cab92a28565" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 37859 } }
I20260925 10:11:58.724401 31989 tablet_bootstrap.cc:492] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba: No bootstrap required, opened a new log
I20260925 10:11:58.724476 31989 ts_tablet_manager.cc:1397] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:11:58.724493 31987 ts_tablet_manager.cc:1428] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:11:58.724620 31970 heartbeater.cc:503] Master 127.14.41.126:44229 was elected leader, sending a full tablet report...
I20260925 10:11:58.725201 31988 raft_consensus.cc:348] T 5c9d7c186a804f19b8f9f1235f33e66e P d2b31ded5b2c485ea0914cab92a28565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413dcea634094f0bbd95f5d6a08c9f6f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34691 } } peers { permanent_uuid: "8be712cca7974a2497a341045de442ba" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38499 } } peers { permanent_uuid: "d2b31ded5b2c485ea0914cab92a28565" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 37859 } }
I20260925 10:11:58.725381 31988 raft_consensus.cc:374] T 5c9d7c186a804f19b8f9f1235f33e66e P d2b31ded5b2c485ea0914cab92a28565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:11:58.725438 31988 raft_consensus.cc:729] T 5c9d7c186a804f19b8f9f1235f33e66e P d2b31ded5b2c485ea0914cab92a28565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2b31ded5b2c485ea0914cab92a28565, State: Initialized, Role: FOLLOWER
I20260925 10:11:58.725553 31988 consensus_queue.cc:260] T 5c9d7c186a804f19b8f9f1235f33e66e P d2b31ded5b2c485ea0914cab92a28565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413dcea634094f0bbd95f5d6a08c9f6f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34691 } } peers { permanent_uuid: "8be712cca7974a2497a341045de442ba" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38499 } } peers { permanent_uuid: "d2b31ded5b2c485ea0914cab92a28565" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 37859 } }
I20260925 10:11:58.725831 31988 ts_tablet_manager.cc:1428] T 5c9d7c186a804f19b8f9f1235f33e66e P d2b31ded5b2c485ea0914cab92a28565: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:11:58.725982 31702 heartbeater.cc:503] Master 127.14.41.126:44229 was elected leader, sending a full tablet report...
I20260925 10:11:58.726677 31989 raft_consensus.cc:348] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413dcea634094f0bbd95f5d6a08c9f6f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34691 } } peers { permanent_uuid: "8be712cca7974a2497a341045de442ba" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38499 } } peers { permanent_uuid: "d2b31ded5b2c485ea0914cab92a28565" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 37859 } }
I20260925 10:11:58.726805 31989 raft_consensus.cc:374] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:11:58.726826 31989 raft_consensus.cc:729] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8be712cca7974a2497a341045de442ba, State: Initialized, Role: FOLLOWER
I20260925 10:11:58.726951 31989 consensus_queue.cc:260] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413dcea634094f0bbd95f5d6a08c9f6f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34691 } } peers { permanent_uuid: "8be712cca7974a2497a341045de442ba" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38499 } } peers { permanent_uuid: "d2b31ded5b2c485ea0914cab92a28565" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 37859 } }
I20260925 10:11:58.727233 31989 ts_tablet_manager.cc:1428] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:11:58.727237 31836 heartbeater.cc:503] Master 127.14.41.126:44229 was elected leader, sending a full tablet report...
W20260925 10:11:58.771680 31837 tablet.cc:2369] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:11:58.777840 31993 raft_consensus.cc:482] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:11:58.777980 31993 raft_consensus.cc:504] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413dcea634094f0bbd95f5d6a08c9f6f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34691 } } peers { permanent_uuid: "8be712cca7974a2497a341045de442ba" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38499 } } peers { permanent_uuid: "d2b31ded5b2c485ea0914cab92a28565" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 37859 } }
I20260925 10:11:58.778455 31993 leader_election.cc:288] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8be712cca7974a2497a341045de442ba (127.14.41.66:38499), d2b31ded5b2c485ea0914cab92a28565 (127.14.41.65:37859)
I20260925 10:11:58.781446 31986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.778752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:50375 (local address 127.14.41.66:38499)
0925 10:11:58.778815 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:58.778817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:58.778830 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:11:58.778863 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:58.778864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:58.778874 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:58.778946 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:11:58.779143 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.779646 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.780663 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.781057 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.781144 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:58.781154 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:58.781336 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 10:11:58.781374 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:11:58.781553 31985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.778886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:42757 (local address 127.14.41.65:37859)
0925 10:11:58.778932 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:58.778934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:58.779008 (+ 74us) server_negotiation.cc:407] Connection header received
0925 10:11:58.779044 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:58.779045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:58.779054 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:58.779106 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:11:58.779351 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.779823 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.780796 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.781182 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.781273 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:58.781281 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:58.781446 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 10:11:58.781487 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:11:58.781567 31996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.778714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:37859 (local address 127.14.41.67:42757)
0925 10:11:58.778928 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:58.778939 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:58.778995 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:58.779107 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:58.779107 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:58.779109 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:11:58.779187 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:58.779189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.779918 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.779919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:58.780687 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:58.780688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.781274 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.781276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:58.781356 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:58.781359 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:11:58.781424 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 10:11:58.781465 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":162,"threads_started":1}
I20260925 10:11:58.781445 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.778666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:38499 (local address 127.14.41.67:50375)
0925 10:11:58.778726 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:58.778739 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:58.778792 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:58.778995 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:58.778996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:58.778998 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:11:58.779059 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:58.779061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.779740 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.779741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:58.780561 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:58.780563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.781141 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.781143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:58.781229 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:58.781232 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:11:58.781325 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 10:11:58.781370 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 10:11:58.781955 31656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c9d7c186a804f19b8f9f1235f33e66e" candidate_uuid: "413dcea634094f0bbd95f5d6a08c9f6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2b31ded5b2c485ea0914cab92a28565" is_pre_election: true
I20260925 10:11:58.782132 31656 raft_consensus.cc:2454] T 5c9d7c186a804f19b8f9f1235f33e66e P d2b31ded5b2c485ea0914cab92a28565 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 413dcea634094f0bbd95f5d6a08c9f6f in term 0.
I20260925 10:11:58.782119 31790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c9d7c186a804f19b8f9f1235f33e66e" candidate_uuid: "413dcea634094f0bbd95f5d6a08c9f6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8be712cca7974a2497a341045de442ba" is_pre_election: true
I20260925 10:11:58.782317 31790 raft_consensus.cc:2454] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 413dcea634094f0bbd95f5d6a08c9f6f in term 0.
I20260925 10:11:58.782369 31857 leader_election.cc:302] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413dcea634094f0bbd95f5d6a08c9f6f, d2b31ded5b2c485ea0914cab92a28565; no voters:
I20260925 10:11:58.782513 31993 raft_consensus.cc:2781] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:11:58.782573 31993 raft_consensus.cc:482] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:11:58.782591 31993 raft_consensus.cc:3037] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:11:58.783368 31993 raft_consensus.cc:504] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413dcea634094f0bbd95f5d6a08c9f6f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34691 } } peers { permanent_uuid: "8be712cca7974a2497a341045de442ba" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38499 } } peers { permanent_uuid: "d2b31ded5b2c485ea0914cab92a28565" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 37859 } }
I20260925 10:11:58.783524 31993 leader_election.cc:288] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [CANDIDATE]: Term 1 election: Requested vote from peers 8be712cca7974a2497a341045de442ba (127.14.41.66:38499), d2b31ded5b2c485ea0914cab92a28565 (127.14.41.65:37859)
I20260925 10:11:58.783664 31790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c9d7c186a804f19b8f9f1235f33e66e" candidate_uuid: "413dcea634094f0bbd95f5d6a08c9f6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8be712cca7974a2497a341045de442ba"
I20260925 10:11:58.783732 31656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c9d7c186a804f19b8f9f1235f33e66e" candidate_uuid: "413dcea634094f0bbd95f5d6a08c9f6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2b31ded5b2c485ea0914cab92a28565"
I20260925 10:11:58.783746 31790 raft_consensus.cc:3037] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:11:58.783816 31656 raft_consensus.cc:3037] T 5c9d7c186a804f19b8f9f1235f33e66e P d2b31ded5b2c485ea0914cab92a28565 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:11:58.784718 31790 raft_consensus.cc:2454] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 413dcea634094f0bbd95f5d6a08c9f6f in term 1.
I20260925 10:11:58.784727 31656 raft_consensus.cc:2454] T 5c9d7c186a804f19b8f9f1235f33e66e P d2b31ded5b2c485ea0914cab92a28565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 413dcea634094f0bbd95f5d6a08c9f6f in term 1.
I20260925 10:11:58.784932 31858 leader_election.cc:302] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 413dcea634094f0bbd95f5d6a08c9f6f, 8be712cca7974a2497a341045de442ba; no voters:
I20260925 10:11:58.785043 31993 raft_consensus.cc:2781] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:11:58.785238 31993 raft_consensus.cc:686] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [term 1 LEADER]: Becoming Leader. State: Replica: 413dcea634094f0bbd95f5d6a08c9f6f, State: Running, Role: LEADER
I20260925 10:11:58.785347 31993 consensus_queue.cc:237] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413dcea634094f0bbd95f5d6a08c9f6f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34691 } } peers { permanent_uuid: "8be712cca7974a2497a341045de442ba" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38499 } } peers { permanent_uuid: "d2b31ded5b2c485ea0914cab92a28565" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 37859 } }
I20260925 10:11:58.786198 31510 catalog_manager.cc:5495] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 413dcea634094f0bbd95f5d6a08c9f6f (127.14.41.67). New cstate: current_term: 1 leader_uuid: "413dcea634094f0bbd95f5d6a08c9f6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "413dcea634094f0bbd95f5d6a08c9f6f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 34691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8be712cca7974a2497a341045de442ba" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2b31ded5b2c485ea0914cab92a28565" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 37859 } health_report { overall_health: UNKNOWN } } }
I20260925 10:11:58.808378 31984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.805564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35642 (local address 127.14.41.67:34691)
0925 10:11:58.805618 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:58.805620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:58.805628 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:11:58.805669 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:58.805670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:58.805690 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:58.805751 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:11:58.805893 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.806421 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.807418 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.807619 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.807700 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:58.807849 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:11:58.808053 (+ 204us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:11:58.808237 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:58.808263 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 10:11:58.808287 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 10:11:58.810600 31656 raft_consensus.cc:1261] T 5c9d7c186a804f19b8f9f1235f33e66e P d2b31ded5b2c485ea0914cab92a28565 [term 1 FOLLOWER]: Refusing update from remote peer 413dcea634094f0bbd95f5d6a08c9f6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:11:58.810627 31790 raft_consensus.cc:1261] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba [term 1 FOLLOWER]: Refusing update from remote peer 413dcea634094f0bbd95f5d6a08c9f6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:11:58.811008 31993 consensus_queue.cc:1035] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2b31ded5b2c485ea0914cab92a28565" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 37859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:11:58.811151 31997 consensus_queue.cc:1035] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8be712cca7974a2497a341045de442ba" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:11:58.813627 32000 mvcc.cc:205] Tried to move back new op lower bound from 7333196262644617216 to 7333196262545510400. Current Snapshot: MvccSnapshot[applied={T|T < 7333196262644617216}]
I20260925 10:11:58.813825 32001 mvcc.cc:205] Tried to move back new op lower bound from 7333196262644617216 to 7333196262545510400. Current Snapshot: MvccSnapshot[applied={T|T < 7333196262644617216}]
I20260925 10:11:58.816342 31509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:11:58.816435 31509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:11:58.817229 31509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04711e99e0734b569e6d7ffd9d33713f: Sending DeleteTablet for 3 replicas of tablet 5c9d7c186a804f19b8f9f1235f33e66e
I20260925 10:11:58.817638 31904 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c9d7c186a804f19b8f9f1235f33e66e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:11:58 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:35640
I20260925 10:11:58.817602 31770 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c9d7c186a804f19b8f9f1235f33e66e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:11:58 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:54582
I20260925 10:11:58.817847 32011 tablet_replica.cc:330] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba: stopping tablet replica
I20260925 10:11:58.817907 31636 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c9d7c186a804f19b8f9f1235f33e66e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:11:58 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:53138
I20260925 10:11:58.817952 32011 raft_consensus.cc:2229] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:11:58.818032 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 31573
I20260925 10:11:58.818064 32011 raft_consensus.cc:2258] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:11:58.818492 32011 ts_tablet_manager.cc:1905] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:11:58.818658 32010 tablet_replica.cc:330] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f: stopping tablet replica
I20260925 10:11:58.818747 32010 raft_consensus.cc:2229] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:11:58.818857 32010 raft_consensus.cc:2258] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:11:58.819334 32010 ts_tablet_manager.cc:1905] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:11:58.820339 32011 ts_tablet_manager.cc:1918] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:11:58.820410 32011 log.cc:1196] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/5c9d7c186a804f19b8f9f1235f33e66e
I20260925 10:11:58.820683 32011 ts_tablet_manager.cc:1939] T 5c9d7c186a804f19b8f9f1235f33e66e P 8be712cca7974a2497a341045de442ba: Deleting consensus metadata
I20260925 10:11:58.821017 32010 ts_tablet_manager.cc:1918] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:11:58.821079 32010 log.cc:1196] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/5c9d7c186a804f19b8f9f1235f33e66e
I20260925 10:11:58.821149 31496 catalog_manager.cc:4841] TS 8be712cca7974a2497a341045de442ba (127.14.41.66:38499): tablet 5c9d7c186a804f19b8f9f1235f33e66e (table test-table [id=779ff0eb384b451eb34d949df0b44c1f]) successfully deleted
I20260925 10:11:58.821354 32010 ts_tablet_manager.cc:1939] T 5c9d7c186a804f19b8f9f1235f33e66e P 413dcea634094f0bbd95f5d6a08c9f6f: Deleting consensus metadata
I20260925 10:11:58.822342 31497 catalog_manager.cc:4841] TS 413dcea634094f0bbd95f5d6a08c9f6f (127.14.41.67:34691): tablet 5c9d7c186a804f19b8f9f1235f33e66e (table test-table [id=779ff0eb384b451eb34d949df0b44c1f]) successfully deleted
W20260925 10:11:58.829046 31495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.41.65:37859 (error 108)
I20260925 10:11:58.829679 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.829524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:37859 (local address 127.0.0.1:53154)
0925 10:11:58.829593 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:58.829635 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.65:37859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 10:11:58.829744 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 31707
I20260925 10:11:58.835827 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 31841
W20260925 10:11:58.835819 31495 catalog_manager.cc:4584] TS d2b31ded5b2c485ea0914cab92a28565 (127.14.41.65:37859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c9d7c186a804f19b8f9f1235f33e66e: Network error: Client connection negotiation failed: client connection to 127.14.41.65:37859: connect: Connection refused (error 111)
I20260925 10:11:58.841079 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 31479
2026-09-25T10:11:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2016 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T10:11:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:11:58Z Disabled control of system clock
I20260925 10:11:58.869575 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:39879
--webserver_interface=127.14.41.126
--webserver_port=0
--builtin_ntp_servers=127.14.41.84:40791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:39879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:11:58.951599 32017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:11:58.951828 32017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:11:58.951850 32017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:11:58.952983 32017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:11:58.953022 32017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:11:58.953035 32017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:11:58.953047 32017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:11:58.953059 32017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:11:58.954344 32017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:39879
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:39879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:11:58.954540 32017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:11:58.954783 32017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:11:58.959820 32023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:58.959942 32017 server_base.cc:1034] running on GCE node
W20260925 10:11:58.959832 32022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:58.959890 32025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:58.960259 32017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:11:58.960511 32017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:11:58.961683 32017 hybrid_clock.cc:648] HybridClock initialized: now 1790331118961669 us; error 43 us; skew 500 ppm
I20260925 10:11:58.962945 32017 webserver.cc:460] Webserver started at http://127.14.41.126:33001/ using document root <none> and password file <none>
I20260925 10:11:58.963310 32017 fs_manager.cc:359] Metadata directory not provided
I20260925 10:11:58.963397 32017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:11:58.963565 32017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:11:58.964880 32017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "6cdabfeeacaf41ada65468372aa13452"
format_stamp: "Formatted at 2026-09-25 10:11:58 on dist-test-slave-s6tm"
server_key: "20431754add891b5e3c39a46a220224a"
server_key_iv: "3adb640c33c9c6cc2ac8a62ed9ba544a"
server_key_version: "encryptionkey@0"
I20260925 10:11:58.965302 32017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "6cdabfeeacaf41ada65468372aa13452"
format_stamp: "Formatted at 2026-09-25 10:11:58 on dist-test-slave-s6tm"
server_key: "20431754add891b5e3c39a46a220224a"
server_key_iv: "3adb640c33c9c6cc2ac8a62ed9ba544a"
server_key_version: "encryptionkey@0"
I20260925 10:11:58.966671 32017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:11:58.967590 32031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:58.967831 32017 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:11:58.967958 32017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "6cdabfeeacaf41ada65468372aa13452"
format_stamp: "Formatted at 2026-09-25 10:11:58 on dist-test-slave-s6tm"
server_key: "20431754add891b5e3c39a46a220224a"
server_key_iv: "3adb640c33c9c6cc2ac8a62ed9ba544a"
server_key_version: "encryptionkey@0"
I20260925 10:11:58.968175 32017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:11:58.980465 32017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:11:58.980849 32017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:11:58.981022 32017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:11:58.985450 32017 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:39879
I20260925 10:11:58.985497 32083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:39879 every 8 connection(s)
I20260925 10:11:58.985878 32017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:11:58.986999 32084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:11:58.989426 32084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452: Bootstrap starting.
I20260925 10:11:58.990072 32084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452: Neither blocks nor log segments found. Creating new log.
I20260925 10:11:58.990381 32084 log.cc:824] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452: Log is configured to *not* fsync() on all Append() calls
I20260925 10:11:58.991060 32084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452: No bootstrap required, opened a new log
I20260925 10:11:58.992754 32084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdabfeeacaf41ada65468372aa13452" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 39879 } }
I20260925 10:11:58.992899 32084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:11:58.992923 32084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cdabfeeacaf41ada65468372aa13452, State: Initialized, Role: FOLLOWER
I20260925 10:11:58.993062 32084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdabfeeacaf41ada65468372aa13452" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 39879 } }
I20260925 10:11:58.993141 32084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:11:58.993165 32084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:11:58.993192 32084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:11:58.993678 32084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdabfeeacaf41ada65468372aa13452" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 39879 } }
I20260925 10:11:58.993775 32084 leader_election.cc:302] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cdabfeeacaf41ada65468372aa13452; no voters:
I20260925 10:11:58.993911 32084 leader_election.cc:288] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:11:58.994001 32087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:11:58.994280 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 32017
I20260925 10:11:58.994432 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:11:58.994625 14501 external_mini_cluster.cc:1428] Setting key 0a693d7e87f2bb9fc9e9b06c880a0860
I20260925 10:11:58.994203 32084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:11:58.994925 32087 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452 [term 1 LEADER]: Becoming Leader. State: Replica: 6cdabfeeacaf41ada65468372aa13452, State: Running, Role: LEADER
I20260925 10:11:58.995215 32087 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdabfeeacaf41ada65468372aa13452" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 39879 } }
I20260925 10:11:58.995682 32089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cdabfeeacaf41ada65468372aa13452. Latest consensus state: current_term: 1 leader_uuid: "6cdabfeeacaf41ada65468372aa13452" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdabfeeacaf41ada65468372aa13452" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 39879 } } }
I20260925 10:11:58.995764 32089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452 [sys.catalog]: This master's current role is: LEADER
I20260925 10:11:58.995882 32088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cdabfeeacaf41ada65468372aa13452" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cdabfeeacaf41ada65468372aa13452" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 39879 } } }
I20260925 10:11:58.995978 32088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452 [sys.catalog]: This master's current role is: LEADER
W20260925 10:11:58.996766 32104 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:11:58.996848 32104 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:11:58.996949 32098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:11:58.997152 32098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:11:59.000789 32098 catalog_manager.cc:1347] Generated new cluster ID: 634efbed1c914e01bc3f844432394398
I20260925 10:11:59.000857 32098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:11:59.001228 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:58.995349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.14.41.126:39879)
0925 10:11:58.995717 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:58.995727 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:58.995762 (+ 35us) server_negotiation.cc:407] Connection header received
0925 10:11:58.995898 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:58.995905 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:58.996021 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:58.996196 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:11:58.996424 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.997741 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.998907 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:58.999324 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:58.999415 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:59.000305 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:11:59.000310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:11:59.000310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:11:59.000512 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:11:59.000548 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:11:59.000560 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:11:59.000647 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:59.000732 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 10:11:59.000841 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":83,"threads_started":1}
I20260925 10:11:59.014201 32098 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:11:59.014953 32098 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:11:59.020123 32098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452: Generated new TSK 0
I20260925 10:11:59.020350 32098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:11:59.026404 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:39879
--builtin_ntp_servers=127.14.41.84:40791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:11:59.110805 32108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:11:59.111070 32108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:11:59.111092 32108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:11:59.112362 32108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:11:59.112401 32108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:11:59.112427 32108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:11:59.113803 32108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:39879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:11:59.114063 32108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:11:59.114373 32108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:11:59.119540 32113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:59.119535 32114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:59.119542 32116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:59.119865 32108 server_base.cc:1034] running on GCE node
I20260925 10:11:59.120123 32108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:11:59.120384 32108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:11:59.121549 32108 hybrid_clock.cc:648] HybridClock initialized: now 1790331119121528 us; error 35 us; skew 500 ppm
I20260925 10:11:59.123075 32108 webserver.cc:460] Webserver started at http://127.14.41.65:39209/ using document root <none> and password file <none>
I20260925 10:11:59.123358 32108 fs_manager.cc:359] Metadata directory not provided
I20260925 10:11:59.123420 32108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:11:59.123549 32108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:11:59.125003 32108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "b8377eee02c8457b9737d527e8f3674e"
format_stamp: "Formatted at 2026-09-25 10:11:59 on dist-test-slave-s6tm"
server_key: "66fc5254457ba0d3b5687212e7887412"
server_key_iv: "7faf88a949fbdf192beaab905bee8806"
server_key_version: "encryptionkey@0"
I20260925 10:11:59.125459 32108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "b8377eee02c8457b9737d527e8f3674e"
format_stamp: "Formatted at 2026-09-25 10:11:59 on dist-test-slave-s6tm"
server_key: "66fc5254457ba0d3b5687212e7887412"
server_key_iv: "7faf88a949fbdf192beaab905bee8806"
server_key_version: "encryptionkey@0"
I20260925 10:11:59.127249 32108 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 10:11:59.128129 32122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:59.128326 32108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:11:59.128412 32108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "b8377eee02c8457b9737d527e8f3674e"
format_stamp: "Formatted at 2026-09-25 10:11:59 on dist-test-slave-s6tm"
server_key: "66fc5254457ba0d3b5687212e7887412"
server_key_iv: "7faf88a949fbdf192beaab905bee8806"
server_key_version: "encryptionkey@0"
I20260925 10:11:59.128479 32108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:11:59.141582 32108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:11:59.141866 32108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:11:59.141969 32108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:11:59.142162 32108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:11:59.142449 32108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:11:59.142480 32108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:59.142503 32108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:11:59.142514 32108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:59.148095 32108 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:40137
I20260925 10:11:59.148191 32235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:40137 every 8 connection(s)
I20260925 10:11:59.148520 32108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:11:59.151562 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 32108
I20260925 10:11:59.151690 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:11:59.151911 14501 external_mini_cluster.cc:1428] Setting key 4cd6787e6f518af99f425838cda25e38
I20260925 10:11:59.152881 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:39879
--builtin_ntp_servers=127.14.41.84:40791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:11:59.154234 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:59.149299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:46789 (local address 127.14.41.126:39879)
0925 10:11:59.149364 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:59.149367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:59.149608 (+ 241us) server_negotiation.cc:407] Connection header received
0925 10:11:59.150124 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:59.150125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:59.150138 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:59.150188 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:11:59.150860 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.151405 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.152630 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.152845 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.152936 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:59.153833 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:11:59.153835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:11:59.153835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:11:59.153841 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:11:59.153857 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:11:59.153862 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:11:59.153933 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:59.154031 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 10:11:59.154065 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 10:11:59.154419 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:59.149289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:39879 (local address 127.14.41.65:46789)
0925 10:11:59.149486 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:59.149521 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:59.150037 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:59.150260 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:59.150264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:59.150272 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:11:59.150792 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:59.150797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.151504 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.151505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:59.152511 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:59.152514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.152948 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.152949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:59.153089 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:59.153689 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:11:59.153690 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:11:59.153719 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:11:59.153742 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:11:59.153933 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:11:59.153940 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:11:59.153944 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:11:59.154022 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 10:11:59.154127 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":104,"threads_started":1}
I20260925 10:11:59.155186 32236 heartbeater.cc:347] Connected to a master server at 127.14.41.126:39879
I20260925 10:11:59.155308 32236 heartbeater.cc:464] Registering TS with master...
I20260925 10:11:59.155507 32236 heartbeater.cc:511] Master 127.14.41.126:39879 requested a full tablet report, sending...
I20260925 10:11:59.156143 32048 ts_manager.cc:194] Registered new tserver with Master: b8377eee02c8457b9737d527e8f3674e (127.14.41.65:40137)
I20260925 10:11:59.156919 32048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.41.65:46789
W20260925 10:11:59.238691 32239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:11:59.238907 32239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:11:59.238930 32239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:11:59.240177 32239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:11:59.240219 32239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:11:59.240245 32239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:11:59.241781 32239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:39879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:11:59.242044 32239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:11:59.242400 32239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:11:59.247618 32244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:59.247689 32245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:59.247673 32239 server_base.cc:1034] running on GCE node
W20260925 10:11:59.247625 32247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:59.248034 32239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:11:59.248313 32239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:11:59.249488 32239 hybrid_clock.cc:648] HybridClock initialized: now 1790331119249483 us; error 40 us; skew 500 ppm
I20260925 10:11:59.250768 32239 webserver.cc:460] Webserver started at http://127.14.41.66:45279/ using document root <none> and password file <none>
I20260925 10:11:59.251029 32239 fs_manager.cc:359] Metadata directory not provided
I20260925 10:11:59.251210 32239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:11:59.251379 32239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:11:59.252549 32239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "8a91f514ef0c40a9abfa78e4b2736606"
format_stamp: "Formatted at 2026-09-25 10:11:59 on dist-test-slave-s6tm"
server_key: "104e35e6a698bd3b1285fcd71e9b3cc2"
server_key_iv: "9904d0f82c2976dc6a57ed5a75d9eb78"
server_key_version: "encryptionkey@0"
I20260925 10:11:59.252913 32239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "8a91f514ef0c40a9abfa78e4b2736606"
format_stamp: "Formatted at 2026-09-25 10:11:59 on dist-test-slave-s6tm"
server_key: "104e35e6a698bd3b1285fcd71e9b3cc2"
server_key_iv: "9904d0f82c2976dc6a57ed5a75d9eb78"
server_key_version: "encryptionkey@0"
I20260925 10:11:59.254290 32239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:11:59.255095 32253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:59.255347 32239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:11:59.255462 32239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "8a91f514ef0c40a9abfa78e4b2736606"
format_stamp: "Formatted at 2026-09-25 10:11:59 on dist-test-slave-s6tm"
server_key: "104e35e6a698bd3b1285fcd71e9b3cc2"
server_key_iv: "9904d0f82c2976dc6a57ed5a75d9eb78"
server_key_version: "encryptionkey@0"
I20260925 10:11:59.255545 32239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:11:59.284511 32239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:11:59.284885 32239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:11:59.285053 32239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:11:59.285315 32239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:11:59.285683 32239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:11:59.285739 32239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:59.285781 32239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:11:59.285823 32239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:59.292114 32239 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:46423
I20260925 10:11:59.292135 32366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:46423 every 8 connection(s)
I20260925 10:11:59.292522 32239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:11:59.298148 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:59.293306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:42389 (local address 127.14.41.126:39879)
0925 10:11:59.293384 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:59.293386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:59.293710 (+ 324us) server_negotiation.cc:407] Connection header received
0925 10:11:59.294308 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:59.294310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:59.294321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:59.294367 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:11:59.295091 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.295730 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.296876 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.297062 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.297118 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:59.297923 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:11:59.297924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:11:59.297924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:11:59.297928 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:11:59.297944 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:11:59.297948 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:11:59.297990 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:59.298062 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 10:11:59.298086 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:11:59.298367 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:59.293335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:39879 (local address 127.14.41.66:42389)
0925 10:11:59.293529 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:59.293613 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:59.294150 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:59.294446 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:59.294450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:59.294459 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:11:59.295019 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:59.295024 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.295829 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.295830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:59.296802 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:59.296805 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.297153 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.297154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:59.297275 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:59.297780 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:11:59.297780 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:11:59.297810 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:11:59.297834 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:11:59.298003 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:11:59.298007 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:11:59.298010 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:11:59.298058 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 10:11:59.298144 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20260925 10:11:59.298882 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 32239
I20260925 10:11:59.298954 32367 heartbeater.cc:347] Connected to a master server at 127.14.41.126:39879
I20260925 10:11:59.298992 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:11:59.299057 32367 heartbeater.cc:464] Registering TS with master...
I20260925 10:11:59.299218 14501 external_mini_cluster.cc:1428] Setting key 3a641fcc8cb2971138afd6fd34b116e8
I20260925 10:11:59.299368 32367 heartbeater.cc:511] Master 127.14.41.126:39879 requested a full tablet report, sending...
I20260925 10:11:59.299831 32047 ts_manager.cc:194] Registered new tserver with Master: 8a91f514ef0c40a9abfa78e4b2736606 (127.14.41.66:46423)
I20260925 10:11:59.300244 32047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.41.66:42389
I20260925 10:11:59.300261 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:39879
--builtin_ntp_servers=127.14.41.84:40791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:11:59.383426 32370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:11:59.383662 32370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:11:59.383687 32370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:11:59.385006 32370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:11:59.385062 32370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:11:59.385113 32370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:11:59.386579 32370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:39879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:11:59.386879 32370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:11:59.387219 32370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:11:59.393136 32378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:59.393045 32375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:11:59.393060 32376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:11:59.393077 32370 server_base.cc:1034] running on GCE node
I20260925 10:11:59.393602 32370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:11:59.393843 32370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:11:59.395009 32370 hybrid_clock.cc:648] HybridClock initialized: now 1790331119394979 us; error 42 us; skew 500 ppm
I20260925 10:11:59.396667 32370 webserver.cc:460] Webserver started at http://127.14.41.67:32775/ using document root <none> and password file <none>
I20260925 10:11:59.396982 32370 fs_manager.cc:359] Metadata directory not provided
I20260925 10:11:59.397032 32370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:11:59.397178 32370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:11:59.398335 32370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "f6000425a41d4d1695c5d0fa8c977b45"
format_stamp: "Formatted at 2026-09-25 10:11:59 on dist-test-slave-s6tm"
server_key: "22d2494682e1969db83400a29f09a2b7"
server_key_iv: "e24310ae237fead26eab06fb00b9019a"
server_key_version: "encryptionkey@0"
I20260925 10:11:59.398725 32370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "f6000425a41d4d1695c5d0fa8c977b45"
format_stamp: "Formatted at 2026-09-25 10:11:59 on dist-test-slave-s6tm"
server_key: "22d2494682e1969db83400a29f09a2b7"
server_key_iv: "e24310ae237fead26eab06fb00b9019a"
server_key_version: "encryptionkey@0"
I20260925 10:11:59.400297 32370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:11:59.401124 32384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:59.401331 32370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:11:59.401433 32370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "f6000425a41d4d1695c5d0fa8c977b45"
format_stamp: "Formatted at 2026-09-25 10:11:59 on dist-test-slave-s6tm"
server_key: "22d2494682e1969db83400a29f09a2b7"
server_key_iv: "e24310ae237fead26eab06fb00b9019a"
server_key_version: "encryptionkey@0"
I20260925 10:11:59.401526 32370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:11:59.410943 32370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:11:59.411324 32370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:11:59.411486 32370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:11:59.411731 32370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:11:59.412160 32370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:11:59.412217 32370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:59.412253 32370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:11:59.412308 32370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:11:59.418401 32370 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:42471
I20260925 10:11:59.418444 32497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:42471 every 8 connection(s)
I20260925 10:11:59.418834 32370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:11:59.424576 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:59.419644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:36963 (local address 127.14.41.126:39879)
0925 10:11:59.419695 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:59.419697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:59.420055 (+ 358us) server_negotiation.cc:407] Connection header received
0925 10:11:59.420637 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:59.420638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:59.420650 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:59.420705 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:11:59.421363 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.421875 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.422883 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.423065 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.423119 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:59.424350 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:11:59.424351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:11:59.424351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:11:59.424356 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:11:59.424369 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:11:59.424372 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:11:59.424415 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:59.424489 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 10:11:59.424513 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:11:59.424796 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:59.419637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:39879 (local address 127.14.41.67:36963)
0925 10:11:59.419882 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:59.419947 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:59.420531 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:59.420762 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:59.420765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:59.420774 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:11:59.421284 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:59.421288 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.421964 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.421965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:59.422800 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:59.422803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.423322 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.423324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:59.423454 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:59.424210 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:11:59.424211 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:11:59.424243 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:11:59.424269 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:11:59.424417 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:11:59.424420 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:11:59.424422 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:11:59.424470 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 10:11:59.424553 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":117,"threads_started":1}
I20260925 10:11:59.425341 32498 heartbeater.cc:347] Connected to a master server at 127.14.41.126:39879
I20260925 10:11:59.425453 32498 heartbeater.cc:464] Registering TS with master...
I20260925 10:11:59.425614 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 32370
I20260925 10:11:59.425669 32498 heartbeater.cc:511] Master 127.14.41.126:39879 requested a full tablet report, sending...
I20260925 10:11:59.425704 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:11:59.425954 14501 external_mini_cluster.cc:1428] Setting key 08f8636ca8cbbcb7921e2a88b523889d
I20260925 10:11:59.426087 32047 ts_manager.cc:194] Registered new tserver with Master: f6000425a41d4d1695c5d0fa8c977b45 (127.14.41.67:42471)
I20260925 10:11:59.426518 32047 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.41.67:36963
I20260925 10:11:59.426774 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:11:59.431247 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:59.428331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45564 (local address 127.14.41.126:39879)
0925 10:11:59.428397 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:59.428399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:59.428500 (+ 101us) server_negotiation.cc:407] Connection header received
0925 10:11:59.428543 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:59.428543 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:59.428554 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:59.428601 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:11:59.428731 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.429173 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.429940 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.430114 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.430168 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:59.430959 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:11:59.430960 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:11:59.430960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:11:59.430965 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:11:59.430978 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:11:59.430981 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:11:59.431023 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:59.431166 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 10:11:59.431187 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 10:11:59.432942 32048 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:11:59.434478 32048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:11:59.440971 32508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:59.436835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49390 (local address 127.14.41.66:46423)
0925 10:11:59.437127 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:59.437132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:59.437143 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:11:59.437302 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:59.437306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:59.437324 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:59.437415 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:11:59.437589 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.438502 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.439836 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.440543 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.440643 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:59.440746 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:59.440841 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 10:11:59.440880 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":103,"threads_started":1}
I20260925 10:11:59.440960 32506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:59.436747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:46423 (local address 127.0.0.1:49390)
0925 10:11:59.437043 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:59.437055 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:59.437115 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:59.437423 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:59.437426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:59.437429 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:11:59.437498 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:59.437500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.438590 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.438591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:59.439734 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:59.439746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.440649 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.440650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:59.440737 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:59.440752 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 10:11:59.440826 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 10:11:59.440880 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":66,"threads_started":1}
I20260925 10:11:59.441519 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:59.437152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47218 (local address 127.14.41.65:40137)
0925 10:11:59.437341 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:59.437345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:59.437475 (+ 130us) server_negotiation.cc:407] Connection header received
0925 10:11:59.437519 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:59.437521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:59.437537 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:59.437609 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:11:59.437768 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.438957 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.440188 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.440953 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.441028 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:59.441114 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:59.441303 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 10:11:59.441348 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":109,"threads_started":1}
I20260925 10:11:59.441862 32509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:59.436933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55738 (local address 127.14.41.67:42471)
0925 10:11:59.437547 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:59.437551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:59.437561 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:11:59.437602 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:59.437604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:59.437619 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:59.437726 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:11:59.437904 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.438957 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.440263 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.440953 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.441538 (+ 585us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:59.441713 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:59.441739 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 10:11:59.441779 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":78,"threads_started":1}
I20260925 10:11:59.441883 32301 tablet_service.cc:1467] Processing CreateTablet for tablet a413588c962c426986b967d9efd79d63 (DEFAULT_TABLE table=test-table [id=a10fdd05a1af45b79163b864d752aef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:11:59.442241 32301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a413588c962c426986b967d9efd79d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:11:59.441375 32510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:59.436952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:40137 (local address 127.0.0.1:47218)
0925 10:11:59.437360 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:59.437370 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:59.437422 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:59.437620 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:59.437622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:59.437623 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:11:59.437683 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:59.437685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.439058 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.439059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:59.440104 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:59.440107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.441148 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.441149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:59.441208 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:59.441212 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:11:59.441262 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 10:11:59.441302 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":74,"threads_started":1}
I20260925 10:11:59.444149 32507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:59.436738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:42471 (local address 127.0.0.1:55738)
0925 10:11:59.436925 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:59.436966 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:59.437029 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:59.437772 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:59.437773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:59.437775 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:11:59.437823 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:59.437824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.439073 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.439074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:59.440183 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:59.440186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.441360 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.441361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:59.441420 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:59.441423 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:11:59.441458 (+ 35us) client_negotiation.cc:240] Negotiation successful
0925 10:11:59.441494 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":89,"thread_start_us":76,"threads_started":1}
I20260925 10:11:59.444818 32170 tablet_service.cc:1467] Processing CreateTablet for tablet a413588c962c426986b967d9efd79d63 (DEFAULT_TABLE table=test-table [id=a10fdd05a1af45b79163b864d752aef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:11:59.444934 32432 tablet_service.cc:1467] Processing CreateTablet for tablet a413588c962c426986b967d9efd79d63 (DEFAULT_TABLE table=test-table [id=a10fdd05a1af45b79163b864d752aef9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:11:59.445189 32170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a413588c962c426986b967d9efd79d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:11:59.445379 32432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a413588c962c426986b967d9efd79d63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:11:59.447552 32513 tablet_bootstrap.cc:492] T a413588c962c426986b967d9efd79d63 P b8377eee02c8457b9737d527e8f3674e: Bootstrap starting.
I20260925 10:11:59.447706 32512 tablet_bootstrap.cc:492] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606: Bootstrap starting.
I20260925 10:11:59.448045 32514 tablet_bootstrap.cc:492] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45: Bootstrap starting.
I20260925 10:11:59.448294 32513 tablet_bootstrap.cc:654] T a413588c962c426986b967d9efd79d63 P b8377eee02c8457b9737d527e8f3674e: Neither blocks nor log segments found. Creating new log.
I20260925 10:11:59.448328 32512 tablet_bootstrap.cc:654] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606: Neither blocks nor log segments found. Creating new log.
I20260925 10:11:59.448648 32512 log.cc:824] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606: Log is configured to *not* fsync() on all Append() calls
I20260925 10:11:59.448649 32513 log.cc:824] T a413588c962c426986b967d9efd79d63 P b8377eee02c8457b9737d527e8f3674e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:11:59.448837 32514 tablet_bootstrap.cc:654] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45: Neither blocks nor log segments found. Creating new log.
I20260925 10:11:59.449210 32514 log.cc:824] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45: Log is configured to *not* fsync() on all Append() calls
I20260925 10:11:59.449461 32513 tablet_bootstrap.cc:492] T a413588c962c426986b967d9efd79d63 P b8377eee02c8457b9737d527e8f3674e: No bootstrap required, opened a new log
I20260925 10:11:59.449461 32512 tablet_bootstrap.cc:492] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606: No bootstrap required, opened a new log
I20260925 10:11:59.449535 32513 ts_tablet_manager.cc:1397] T a413588c962c426986b967d9efd79d63 P b8377eee02c8457b9737d527e8f3674e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:11:59.449535 32512 ts_tablet_manager.cc:1397] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 10:11:59.449892 32514 tablet_bootstrap.cc:492] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45: No bootstrap required, opened a new log
I20260925 10:11:59.449978 32514 ts_tablet_manager.cc:1397] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 10:11:59.451807 32514 raft_consensus.cc:348] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6000425a41d4d1695c5d0fa8c977b45" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42471 } } peers { permanent_uuid: "8a91f514ef0c40a9abfa78e4b2736606" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 46423 } } peers { permanent_uuid: "b8377eee02c8457b9737d527e8f3674e" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40137 } }
I20260925 10:11:59.451998 32514 raft_consensus.cc:374] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:11:59.452082 32514 raft_consensus.cc:729] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6000425a41d4d1695c5d0fa8c977b45, State: Initialized, Role: FOLLOWER
I20260925 10:11:59.452256 32513 raft_consensus.cc:348] T a413588c962c426986b967d9efd79d63 P b8377eee02c8457b9737d527e8f3674e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6000425a41d4d1695c5d0fa8c977b45" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42471 } } peers { permanent_uuid: "8a91f514ef0c40a9abfa78e4b2736606" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 46423 } } peers { permanent_uuid: "b8377eee02c8457b9737d527e8f3674e" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40137 } }
I20260925 10:11:59.452262 32512 raft_consensus.cc:348] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6000425a41d4d1695c5d0fa8c977b45" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42471 } } peers { permanent_uuid: "8a91f514ef0c40a9abfa78e4b2736606" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 46423 } } peers { permanent_uuid: "b8377eee02c8457b9737d527e8f3674e" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40137 } }
I20260925 10:11:59.452239 32514 consensus_queue.cc:260] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6000425a41d4d1695c5d0fa8c977b45" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42471 } } peers { permanent_uuid: "8a91f514ef0c40a9abfa78e4b2736606" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 46423 } } peers { permanent_uuid: "b8377eee02c8457b9737d527e8f3674e" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40137 } }
I20260925 10:11:59.452421 32512 raft_consensus.cc:374] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:11:59.452482 32512 raft_consensus.cc:729] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a91f514ef0c40a9abfa78e4b2736606, State: Initialized, Role: FOLLOWER
I20260925 10:11:59.452556 32513 raft_consensus.cc:374] T a413588c962c426986b967d9efd79d63 P b8377eee02c8457b9737d527e8f3674e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:11:59.452653 32513 raft_consensus.cc:729] T a413588c962c426986b967d9efd79d63 P b8377eee02c8457b9737d527e8f3674e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8377eee02c8457b9737d527e8f3674e, State: Initialized, Role: FOLLOWER
I20260925 10:11:59.452646 32514 ts_tablet_manager.cc:1428] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 10:11:59.452638 32512 consensus_queue.cc:260] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6000425a41d4d1695c5d0fa8c977b45" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42471 } } peers { permanent_uuid: "8a91f514ef0c40a9abfa78e4b2736606" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 46423 } } peers { permanent_uuid: "b8377eee02c8457b9737d527e8f3674e" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40137 } }
I20260925 10:11:59.452709 32498 heartbeater.cc:503] Master 127.14.41.126:39879 was elected leader, sending a full tablet report...
I20260925 10:11:59.452898 32512 ts_tablet_manager.cc:1428] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260925 10:11:59.452859 32513 consensus_queue.cc:260] T a413588c962c426986b967d9efd79d63 P b8377eee02c8457b9737d527e8f3674e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6000425a41d4d1695c5d0fa8c977b45" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42471 } } peers { permanent_uuid: "8a91f514ef0c40a9abfa78e4b2736606" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 46423 } } peers { permanent_uuid: "b8377eee02c8457b9737d527e8f3674e" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40137 } }
I20260925 10:11:59.452930 32367 heartbeater.cc:503] Master 127.14.41.126:39879 was elected leader, sending a full tablet report...
I20260925 10:11:59.453106 32513 ts_tablet_manager.cc:1428] T a413588c962c426986b967d9efd79d63 P b8377eee02c8457b9737d527e8f3674e: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260925 10:11:59.453169 32236 heartbeater.cc:503] Master 127.14.41.126:39879 was elected leader, sending a full tablet report...
I20260925 10:11:59.476589 32518 raft_consensus.cc:482] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:11:59.476750 32518 raft_consensus.cc:504] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6000425a41d4d1695c5d0fa8c977b45" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42471 } } peers { permanent_uuid: "8a91f514ef0c40a9abfa78e4b2736606" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 46423 } } peers { permanent_uuid: "b8377eee02c8457b9737d527e8f3674e" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40137 } }
I20260925 10:11:59.477169 32518 leader_election.cc:288] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a91f514ef0c40a9abfa78e4b2736606 (127.14.41.66:46423), b8377eee02c8457b9737d527e8f3674e (127.14.41.65:40137)
I20260925 10:11:59.480330 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:59.477594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:37469 (local address 127.14.41.65:40137)
0925 10:11:59.477665 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:59.477667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:59.477810 (+ 143us) server_negotiation.cc:407] Connection header received
0925 10:11:59.477840 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:59.477842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:59.477855 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:59.477911 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:11:59.478073 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.478530 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.479554 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.479917 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.480037 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:59.480046 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:59.480204 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 10:11:59.480247 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:11:59.480340 32508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:59.477458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:52549 (local address 127.14.41.66:46423)
0925 10:11:59.477537 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:59.477540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:59.477551 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:11:59.477593 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:59.477595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:59.477605 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:59.477674 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:11:59.477907 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.478502 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.479519 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.479903 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.480042 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:59.480051 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:59.480203 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 10:11:59.480250 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 10:11:59.480369 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:59.477358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:46423 (local address 127.14.41.67:52549)
0925 10:11:59.477450 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:59.477464 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:59.477524 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:59.477689 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:59.477689 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:59.477691 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:11:59.477793 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:59.477796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.478601 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.478603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:59.479415 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:59.479418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.480047 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.480048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:59.480124 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:59.480127 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:11:59.480197 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 10:11:59.480267 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":21}
I20260925 10:11:59.480386 32521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:59.477514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:40137 (local address 127.14.41.67:37469)
0925 10:11:59.477695 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:59.477704 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:11:59.477765 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:11:59.477924 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:11:59.477924 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:11:59.477926 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:11:59.477999 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:59.478000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.478649 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.478649 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:59.479427 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:11:59.479429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.480038 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.480039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:11:59.480116 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:59.480125 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 10:11:59.480196 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 10:11:59.480264 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":3,"thread_start_us":116,"threads_started":1}
I20260925 10:11:59.480804 32190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a413588c962c426986b967d9efd79d63" candidate_uuid: "f6000425a41d4d1695c5d0fa8c977b45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8377eee02c8457b9737d527e8f3674e" is_pre_election: true
I20260925 10:11:59.480774 32321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a413588c962c426986b967d9efd79d63" candidate_uuid: "f6000425a41d4d1695c5d0fa8c977b45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a91f514ef0c40a9abfa78e4b2736606" is_pre_election: true
I20260925 10:11:59.480958 32190 raft_consensus.cc:2454] T a413588c962c426986b967d9efd79d63 P b8377eee02c8457b9737d527e8f3674e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6000425a41d4d1695c5d0fa8c977b45 in term 0.
I20260925 10:11:59.480958 32321 raft_consensus.cc:2454] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6000425a41d4d1695c5d0fa8c977b45 in term 0.
I20260925 10:11:59.481238 32388 leader_election.cc:302] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8377eee02c8457b9737d527e8f3674e, f6000425a41d4d1695c5d0fa8c977b45; no voters:
I20260925 10:11:59.481426 32518 raft_consensus.cc:2781] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:11:59.481513 32518 raft_consensus.cc:482] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:11:59.481531 32518 raft_consensus.cc:3037] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:11:59.482445 32518 raft_consensus.cc:504] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6000425a41d4d1695c5d0fa8c977b45" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42471 } } peers { permanent_uuid: "8a91f514ef0c40a9abfa78e4b2736606" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 46423 } } peers { permanent_uuid: "b8377eee02c8457b9737d527e8f3674e" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40137 } }
I20260925 10:11:59.482620 32518 leader_election.cc:288] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [CANDIDATE]: Term 1 election: Requested vote from peers 8a91f514ef0c40a9abfa78e4b2736606 (127.14.41.66:46423), b8377eee02c8457b9737d527e8f3674e (127.14.41.65:40137)
I20260925 10:11:59.482793 32190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a413588c962c426986b967d9efd79d63" candidate_uuid: "f6000425a41d4d1695c5d0fa8c977b45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8377eee02c8457b9737d527e8f3674e"
I20260925 10:11:59.482789 32321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a413588c962c426986b967d9efd79d63" candidate_uuid: "f6000425a41d4d1695c5d0fa8c977b45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a91f514ef0c40a9abfa78e4b2736606"
I20260925 10:11:59.482921 32190 raft_consensus.cc:3037] T a413588c962c426986b967d9efd79d63 P b8377eee02c8457b9737d527e8f3674e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:11:59.482924 32321 raft_consensus.cc:3037] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:11:59.483773 32190 raft_consensus.cc:2454] T a413588c962c426986b967d9efd79d63 P b8377eee02c8457b9737d527e8f3674e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6000425a41d4d1695c5d0fa8c977b45 in term 1.
I20260925 10:11:59.483775 32321 raft_consensus.cc:2454] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6000425a41d4d1695c5d0fa8c977b45 in term 1.
I20260925 10:11:59.483978 32388 leader_election.cc:302] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a91f514ef0c40a9abfa78e4b2736606, f6000425a41d4d1695c5d0fa8c977b45; no voters:
I20260925 10:11:59.484067 32518 raft_consensus.cc:2781] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:11:59.484221 32518 raft_consensus.cc:686] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [term 1 LEADER]: Becoming Leader. State: Replica: f6000425a41d4d1695c5d0fa8c977b45, State: Running, Role: LEADER
I20260925 10:11:59.484339 32518 consensus_queue.cc:237] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6000425a41d4d1695c5d0fa8c977b45" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42471 } } peers { permanent_uuid: "8a91f514ef0c40a9abfa78e4b2736606" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 46423 } } peers { permanent_uuid: "b8377eee02c8457b9737d527e8f3674e" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40137 } }
I20260925 10:11:59.485334 32048 catalog_manager.cc:5495] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6000425a41d4d1695c5d0fa8c977b45 (127.14.41.67). New cstate: current_term: 1 leader_uuid: "f6000425a41d4d1695c5d0fa8c977b45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6000425a41d4d1695c5d0fa8c977b45" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a91f514ef0c40a9abfa78e4b2736606" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 46423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8377eee02c8457b9737d527e8f3674e" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40137 } health_report { overall_health: UNKNOWN } } }
I20260925 10:11:59.499645 32509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:59.497226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.14.41.67:42471)
0925 10:11:59.497298 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:11:59.497300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:11:59.497309 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:11:59.497355 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:11:59.497356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:11:59.497373 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:11:59.497452 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:11:59.497561 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.498041 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.498818 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:11:59.499022 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:11:59.499099 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:11:59.499286 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:11:59.499395 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:11:59.499492 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 10:11:59.499511 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 10:11:59.499529 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:11:59.501912 32321 raft_consensus.cc:1261] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606 [term 1 FOLLOWER]: Refusing update from remote peer f6000425a41d4d1695c5d0fa8c977b45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:11:59.501912 32190 raft_consensus.cc:1261] T a413588c962c426986b967d9efd79d63 P b8377eee02c8457b9737d527e8f3674e [term 1 FOLLOWER]: Refusing update from remote peer f6000425a41d4d1695c5d0fa8c977b45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:11:59.502331 32522 consensus_queue.cc:1035] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a91f514ef0c40a9abfa78e4b2736606" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 46423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:11:59.502481 32518 consensus_queue.cc:1035] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8377eee02c8457b9737d527e8f3674e" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 40137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:11:59.505052 32525 mvcc.cc:205] Tried to move back new op lower bound from 7333196265476030464 to 7333196265408679936. Current Snapshot: MvccSnapshot[applied={T|T < 7333196265476030464}]
I20260925 10:11:59.505916 32527 mvcc.cc:205] Tried to move back new op lower bound from 7333196265476030464 to 7333196265408679936. Current Snapshot: MvccSnapshot[applied={T|T < 7333196265476030464}]
I20260925 10:11:59.506397 32526 mvcc.cc:205] Tried to move back new op lower bound from 7333196265476030464 to 7333196265408679936. Current Snapshot: MvccSnapshot[applied={T|T < 7333196265476030464}]
I20260925 10:11:59.507627 32048 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:11:59.507776 32048 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:11:59.508615 32048 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6cdabfeeacaf41ada65468372aa13452: Sending DeleteTablet for 3 replicas of tablet a413588c962c426986b967d9efd79d63
I20260925 10:11:59.508922 32432 tablet_service.cc:1514] Processing DeleteTablet for tablet a413588c962c426986b967d9efd79d63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:11:59 UTC) from {username='slave'} at 127.0.0.1:55738
I20260925 10:11:59.508977 32170 tablet_service.cc:1514] Processing DeleteTablet for tablet a413588c962c426986b967d9efd79d63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:11:59 UTC) from {username='slave'} at 127.0.0.1:47218
I20260925 10:11:59.509268 32534 tablet_replica.cc:330] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45: stopping tablet replica
I20260925 10:11:59.509352 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 32108
I20260925 10:11:59.509358 32301 tablet_service.cc:1514] Processing DeleteTablet for tablet a413588c962c426986b967d9efd79d63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:11:59 UTC) from {username='slave'} at 127.0.0.1:49390
I20260925 10:11:59.509421 32534 raft_consensus.cc:2229] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:11:59.509549 32534 raft_consensus.cc:2258] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:11:59.510042 32534 ts_tablet_manager.cc:1905] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:11:59.511698 32536 tablet_replica.cc:330] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606: stopping tablet replica
I20260925 10:11:59.511895 32536 raft_consensus.cc:2229] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:11:59.512090 32536 raft_consensus.cc:2258] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:11:59.512456 32534 ts_tablet_manager.cc:1918] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:11:59.512579 32534 log.cc:1196] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/a413588c962c426986b967d9efd79d63
I20260925 10:11:59.512588 32536 ts_tablet_manager.cc:1905] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:11:59.512955 32534 ts_tablet_manager.cc:1939] T a413588c962c426986b967d9efd79d63 P f6000425a41d4d1695c5d0fa8c977b45: Deleting consensus metadata
I20260925 10:11:59.513448 32034 catalog_manager.cc:4841] TS f6000425a41d4d1695c5d0fa8c977b45 (127.14.41.67:42471): tablet a413588c962c426986b967d9efd79d63 (table test-table [id=a10fdd05a1af45b79163b864d752aef9]) successfully deleted
I20260925 10:11:59.514187 32536 ts_tablet_manager.cc:1918] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:11:59.514277 32536 log.cc:1196] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/a413588c962c426986b967d9efd79d63
I20260925 10:11:59.514519 32536 ts_tablet_manager.cc:1939] T a413588c962c426986b967d9efd79d63 P 8a91f514ef0c40a9abfa78e4b2736606: Deleting consensus metadata
I20260925 10:11:59.515228 32035 catalog_manager.cc:4841] TS 8a91f514ef0c40a9abfa78e4b2736606 (127.14.41.66:46423): tablet a413588c962c426986b967d9efd79d63 (table test-table [id=a10fdd05a1af45b79163b864d752aef9]) successfully deleted
W20260925 10:11:59.516216 32035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.41.65:40137 (error 108)
I20260925 10:11:59.516467 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 32239
I20260925 10:11:59.517179 32507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:11:59.516645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:40137 (local address 127.0.0.1:47226)
0925 10:11:59.517091 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 10:11:59.517137 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.65:40137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419}
W20260925 10:11:59.517315 32035 catalog_manager.cc:4584] TS b8377eee02c8457b9737d527e8f3674e (127.14.41.65:40137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a413588c962c426986b967d9efd79d63: Network error: Client connection negotiation failed: client connection to 127.14.41.65:40137: connect: Connection refused (error 111)
I20260925 10:11:59.522126 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 32370
I20260925 10:11:59.527184 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 32017
2026-09-25T10:11:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (685 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:11:59 dist-test-slave-s6tm krb5kdc[32540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:11:59 dist-test-slave-s6tm krb5kdc[32540](info): set up 2 sockets
Sep 25 10:11:59 dist-test-slave-s6tm krb5kdc[32540](info): commencing operation
krb5kdc: starting...
W20260925 10:12:00.577082 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:00 dist-test-slave-s6tm krb5kdc[32540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:12:00.702348 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:45273
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:45273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:00.785068 32553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:00.785327 32553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:00.785369 32553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:00.786536 32553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:00.786605 32553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:00.786629 32553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:00.786652 32553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:00.786680 32553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:00.788192 32553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:45273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:45273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:00.788484 32553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:00.788712 32553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:00.793629 32559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:00.793649 32561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:00.793678 32558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:00.794317 32553 server_base.cc:1034] running on GCE node
I20260925 10:12:00.794590 32553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:00.794927 32553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:00.796089 32553 hybrid_clock.cc:648] HybridClock initialized: now 1790331120796069 us; error 31 us; skew 500 ppm
Sep 25 10:12:00 dist-test-slave-s6tm krb5kdc[32540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331120, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:00.798738 32553 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:12:00.799247 32553 webserver.cc:460] Webserver started at http://127.14.41.126:37737/ using document root <none> and password file <none>
I20260925 10:12:00.799491 32553 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:00.799562 32553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:00.799726 32553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:00.800894 32553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "127ca165b0794f48a91b37b5eca6c695"
format_stamp: "Formatted at 2026-09-25 10:12:00 on dist-test-slave-s6tm"
server_key: "65c542331d07ca2d7743244c27a8b185"
server_key_iv: "80845e8668253e343f8f10bcd234d193"
server_key_version: "encryptionkey@0"
I20260925 10:12:00.801270 32553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "127ca165b0794f48a91b37b5eca6c695"
format_stamp: "Formatted at 2026-09-25 10:12:00 on dist-test-slave-s6tm"
server_key: "65c542331d07ca2d7743244c27a8b185"
server_key_iv: "80845e8668253e343f8f10bcd234d193"
server_key_version: "encryptionkey@0"
I20260925 10:12:00.802627 32553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:12:00.803375 32568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:00.803555 32553 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:12:00.803654 32553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "127ca165b0794f48a91b37b5eca6c695"
format_stamp: "Formatted at 2026-09-25 10:12:00 on dist-test-slave-s6tm"
server_key: "65c542331d07ca2d7743244c27a8b185"
server_key_iv: "80845e8668253e343f8f10bcd234d193"
server_key_version: "encryptionkey@0"
I20260925 10:12:00.803745 32553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:00.819144 32553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:00.821136 32553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:00.821310 32553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:00.825460 32553 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:45273
I20260925 10:12:00.825495 32620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:45273 every 8 connection(s)
I20260925 10:12:00.825875 32553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:00.826757 32621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:00.827348 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 32553
I20260925 10:12:00.827489 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:00.827700 14501 external_mini_cluster.cc:1428] Setting key 4fef6819372de0075d690e660d829baf
I20260925 10:12:00.829881 32621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695: Bootstrap starting.
I20260925 10:12:00.830626 32621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:00.830945 32621 log.cc:824] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:00.832358 32621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695: No bootstrap required, opened a new log
Sep 25 10:12:00 dist-test-slave-s6tm krb5kdc[32540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:12:00.835060 32621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127ca165b0794f48a91b37b5eca6c695" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45273 } }
I20260925 10:12:00.835256 32621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:00.835282 32621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 127ca165b0794f48a91b37b5eca6c695, State: Initialized, Role: FOLLOWER
I20260925 10:12:00.835420 32621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127ca165b0794f48a91b37b5eca6c695" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45273 } }
I20260925 10:12:00.835491 32621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:00.835512 32621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:00.835611 32621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:00.836269 32621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127ca165b0794f48a91b37b5eca6c695" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45273 } }
I20260925 10:12:00.836391 32621 leader_election.cc:302] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 127ca165b0794f48a91b37b5eca6c695; no voters:
I20260925 10:12:00.836587 32621 leader_election.cc:288] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:00.836622 32626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:00.836771 32626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695 [term 1 LEADER]: Becoming Leader. State: Replica: 127ca165b0794f48a91b37b5eca6c695, State: Running, Role: LEADER
I20260925 10:12:00.836877 32621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:00.836938 32626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127ca165b0794f48a91b37b5eca6c695" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45273 } }
I20260925 10:12:00.837298 32627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 127ca165b0794f48a91b37b5eca6c695. Latest consensus state: current_term: 1 leader_uuid: "127ca165b0794f48a91b37b5eca6c695" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127ca165b0794f48a91b37b5eca6c695" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45273 } } }
I20260925 10:12:00.837527 32627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:00.837977 32628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "127ca165b0794f48a91b37b5eca6c695" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "127ca165b0794f48a91b37b5eca6c695" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 45273 } } }
I20260925 10:12:00.838066 32628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:00.838198 32635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:00.838830 32635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:00.839715 32624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:00.828257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.14.41.126:45273)
0925 10:12:00.828467 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:00.828477 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:00.828512 (+ 35us) server_negotiation.cc:407] Connection header received
0925 10:12:00.829123 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:00.829133 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:00.829200 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:00.829353 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:00.829978 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:00.831329 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:00.832238 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:00.832706 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:00.832817 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:00.835619 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:00.835622 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:00.835622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:00.835632 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:00.837851 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:00.838355 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:00.838358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:00.838359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:00.838479 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:00.838719 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:00.838720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:00.838720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:00.839032 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:00.839194 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:00.839372 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 10:12:00.839517 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":94,"threads_started":1}
I20260925 10:12:00.840451 32635 catalog_manager.cc:1347] Generated new cluster ID: 5468fc5443be4e93bef0c95935aa3feb
I20260925 10:12:00.840500 32635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:00.855244 32635 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:00.856413 32635 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:00.859659 32635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695: Generated new TSK 0
I20260925 10:12:00.859898 32635 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
I20260925 10:12:00.891443 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:45273
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:00.976727 32647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:00.976939 32647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:00.976965 32647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:00.978240 32647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:00.978293 32647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:00.978324 32647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:12:00.979911 32647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:45273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:00.980239 32647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:00.980515 32647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:12:00.986007 32647 server_base.cc:1034] running on GCE node
W20260925 10:12:00.985986 32652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:00.985975 32653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:00.985988 32655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:00.986460 32647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:00.986691 32647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:00.987852 32647 hybrid_clock.cc:648] HybridClock initialized: now 1790331120987837 us; error 29 us; skew 500 ppm
Sep 25 10:12:00 dist-test-slave-s6tm krb5kdc[32540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331120, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:00.990782 32647 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:12:00.991362 32647 webserver.cc:460] Webserver started at http://127.14.41.65:40609/ using document root <none> and password file <none>
I20260925 10:12:00.991628 32647 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:00.991703 32647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:00.991853 32647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:00.992972 32647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "8bd6369556e145408e4a53eab11ef14d"
format_stamp: "Formatted at 2026-09-25 10:12:00 on dist-test-slave-s6tm"
server_key: "457d0de712e150ededf7726135a65337"
server_key_iv: "20443b3ed3baec29650d7073ad3e5402"
server_key_version: "encryptionkey@0"
I20260925 10:12:00.993364 32647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "8bd6369556e145408e4a53eab11ef14d"
format_stamp: "Formatted at 2026-09-25 10:12:00 on dist-test-slave-s6tm"
server_key: "457d0de712e150ededf7726135a65337"
server_key_iv: "20443b3ed3baec29650d7073ad3e5402"
server_key_version: "encryptionkey@0"
I20260925 10:12:00.994900 32647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:12:00.995775 32662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:00.996006 32647 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:00.996112 32647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "8bd6369556e145408e4a53eab11ef14d"
format_stamp: "Formatted at 2026-09-25 10:12:00 on dist-test-slave-s6tm"
server_key: "457d0de712e150ededf7726135a65337"
server_key_iv: "20443b3ed3baec29650d7073ad3e5402"
server_key_version: "encryptionkey@0"
I20260925 10:12:00.996209 32647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:01.004122 32647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:01.006749 32647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:01.006929 32647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:01.007180 32647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:01.007552 32647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:01.007619 32647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:01.007665 32647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:01.007692 32647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:01.013995 32647 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:42279
I20260925 10:12:01.014045 307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:42279 every 8 connection(s)
I20260925 10:12:01.014411 32647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:12:01.016953 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 32647
I20260925 10:12:01.017087 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:12:01.017300 14501 external_mini_cluster.cc:1428] Setting key 6f5727cd38cb7ac7c7dd584b1f8c791d
Sep 25 10:12:01 dist-test-slave-s6tm krb5kdc[32540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331120, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:12:01.025388 32624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.015213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:58707 (local address 127.14.41.126:45273)
0925 10:12:01.015281 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:01.015284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:01.015504 (+ 220us) server_negotiation.cc:407] Connection header received
0925 10:12:01.016043 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:01.016044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:01.016055 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:01.016110 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:01.017099 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.017772 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.019335 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.019575 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.019665 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.022149 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:01.022151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:01.022152 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:01.022160 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:01.023754 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:01.024320 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:01.024322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:01.024322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:01.024389 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:01.024665 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:01.024666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:01.024666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:01.024802 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:01.024882 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:01.025179 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 10:12:01.025288 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 10:12:01.025646 310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.015204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:45273 (local address 127.14.41.65:58707)
0925 10:12:01.015380 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:01.015418 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:01.015902 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:01.016169 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:01.016172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:01.016786 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:01.016987 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:01.016992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.017868 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.017869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:01.019237 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:01.019241 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.019677 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.019678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:01.019821 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.020456 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:01.020466 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:01.022026 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:01.023874 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:01.023882 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:01.023885 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:01.024185 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:01.024460 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:01.024461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:01.024461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:01.024585 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:01.024872 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:01.024879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:01.025037 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 10:12:01.025146 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 10:12:01.025330 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.14.41.66@KRBTEST.COM; defaulting to no policy
I20260925 10:12:01.026125 308 heartbeater.cc:347] Connected to a master server at 127.14.41.126:45273
I20260925 10:12:01.026227 308 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:01.026455 308 heartbeater.cc:511] Master 127.14.41.126:45273 requested a full tablet report, sending...
I20260925 10:12:01.027019 32585 ts_manager.cc:194] Registered new tserver with Master: 8bd6369556e145408e4a53eab11ef14d (127.14.41.65:42279)
I20260925 10:12:01.027678 32585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:58707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
I20260925 10:12:01.043819 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:45273
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:01.128268 313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:01.128567 313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:01.128597 313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:01.129738 313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:01.129776 313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:01.129805 313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:12:01.131242 313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:45273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:01.131444 313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:01.131623 313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:12:01.136904 313 server_base.cc:1034] running on GCE node
W20260925 10:12:01.136886 321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:01.137080 318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:01.136879 319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:01.137326 313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:01.137624 313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:01.138826 313 hybrid_clock.cc:648] HybridClock initialized: now 1790331121138812 us; error 31 us; skew 500 ppm
Sep 25 10:12:01 dist-test-slave-s6tm krb5kdc[32540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331121, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:01.146960 313 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:12:01.147495 313 webserver.cc:460] Webserver started at http://127.14.41.66:43311/ using document root <none> and password file <none>
I20260925 10:12:01.147755 313 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:01.147827 313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:01.148020 313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:01.149250 313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "3c0fd31573364accafd38b0819d11e12"
format_stamp: "Formatted at 2026-09-25 10:12:01 on dist-test-slave-s6tm"
server_key: "9a51476b2a387bb895960e2785fc333a"
server_key_iv: "bb4882af9e7d95f515209a564a338468"
server_key_version: "encryptionkey@0"
I20260925 10:12:01.149624 313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "3c0fd31573364accafd38b0819d11e12"
format_stamp: "Formatted at 2026-09-25 10:12:01 on dist-test-slave-s6tm"
server_key: "9a51476b2a387bb895960e2785fc333a"
server_key_iv: "bb4882af9e7d95f515209a564a338468"
server_key_version: "encryptionkey@0"
I20260925 10:12:01.151031 313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:12:01.151849 328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:01.152038 313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:01.152164 313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "3c0fd31573364accafd38b0819d11e12"
format_stamp: "Formatted at 2026-09-25 10:12:01 on dist-test-slave-s6tm"
server_key: "9a51476b2a387bb895960e2785fc333a"
server_key_iv: "bb4882af9e7d95f515209a564a338468"
server_key_version: "encryptionkey@0"
I20260925 10:12:01.152245 313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:01.183111 313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:01.185410 313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:01.185608 313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:01.185868 313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:01.186221 313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:01.186281 313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:01.186324 313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:01.186354 313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:01.192864 313 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:42591
I20260925 10:12:01.192905 441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:42591 every 8 connection(s)
I20260925 10:12:01.193256 313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:12:01 dist-test-slave-s6tm krb5kdc[32540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331121, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:12:01.199779 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 313
I20260925 10:12:01.199913 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:12:01.200150 14501 external_mini_cluster.cc:1428] Setting key b07b6d4100125192bfbc240dafd61910
I20260925 10:12:01.203656 32624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.193990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:54597 (local address 127.14.41.126:45273)
0925 10:12:01.194067 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:01.194069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:01.194292 (+ 223us) server_negotiation.cc:407] Connection header received
0925 10:12:01.194867 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:01.194868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:01.194883 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:01.194934 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:01.195973 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.196507 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.197547 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.197776 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.197854 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.200276 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:01.200278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:01.200279 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:01.200288 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:01.202101 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:01.202560 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:01.202561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:01.202562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:01.202636 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:01.202990 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:01.202991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:01.202991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:01.203102 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:01.203156 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:01.203515 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 10:12:01.203589 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 10:12:01.203969 444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.193990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:45273 (local address 127.14.41.66:54597)
0925 10:12:01.194164 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:01.194204 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:01.194736 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:01.195019 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:01.195024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:01.195634 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:01.195849 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:01.195854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.196625 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.196626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:01.197451 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:01.197453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.197875 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.197877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:01.198002 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.198550 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:01.198558 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:01.200133 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:01.202199 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:01.202207 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:01.202209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:01.202473 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:01.202723 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:01.202724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:01.202724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:01.202916 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:01.203174 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:01.203180 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:01.203365 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 10:12:01.203490 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 10:12:01.203691 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260925 10:12:01.204485 442 heartbeater.cc:347] Connected to a master server at 127.14.41.126:45273
I20260925 10:12:01.204603 442 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:01.204814 442 heartbeater.cc:511] Master 127.14.41.126:45273 requested a full tablet report, sending...
I20260925 10:12:01.205291 32585 ts_manager.cc:194] Registered new tserver with Master: 3c0fd31573364accafd38b0819d11e12 (127.14.41.66:42591)
I20260925 10:12:01.205690 32585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:54597
WARNING: no policy specified for kudu/127.14.41.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
I20260925 10:12:01.227092 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:45273
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:40013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:01.313890 447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:01.314139 447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:01.314167 447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:01.315371 447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:01.315411 447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:01.315439 447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:12:01.317085 447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:45273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:01.317310 447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:01.317520 447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:01.322876 452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:01.322855 453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:01.322846 455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:01.322923 447 server_base.cc:1034] running on GCE node
I20260925 10:12:01.323278 447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:01.323536 447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:01.324702 447 hybrid_clock.cc:648] HybridClock initialized: now 1790331121324688 us; error 49 us; skew 500 ppm
Sep 25 10:12:01 dist-test-slave-s6tm krb5kdc[32540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331121, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:01.327307 447 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:12:01.327765 447 webserver.cc:460] Webserver started at http://127.14.41.67:42213/ using document root <none> and password file <none>
I20260925 10:12:01.328055 447 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:01.328128 447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:01.328295 447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:01.329381 447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "c6a7a466e4be42c1b0a55229068023b5"
format_stamp: "Formatted at 2026-09-25 10:12:01 on dist-test-slave-s6tm"
server_key: "a7021ee9af83158784cc90d27a498473"
server_key_iv: "3ec2bfa8a25f4fec840b069787beb5be"
server_key_version: "encryptionkey@0"
I20260925 10:12:01.329766 447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "c6a7a466e4be42c1b0a55229068023b5"
format_stamp: "Formatted at 2026-09-25 10:12:01 on dist-test-slave-s6tm"
server_key: "a7021ee9af83158784cc90d27a498473"
server_key_iv: "3ec2bfa8a25f4fec840b069787beb5be"
server_key_version: "encryptionkey@0"
I20260925 10:12:01.331139 447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:12:01.331976 462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:01.332185 447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:12:01.332342 447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "c6a7a466e4be42c1b0a55229068023b5"
format_stamp: "Formatted at 2026-09-25 10:12:01 on dist-test-slave-s6tm"
server_key: "a7021ee9af83158784cc90d27a498473"
server_key_iv: "3ec2bfa8a25f4fec840b069787beb5be"
server_key_version: "encryptionkey@0"
I20260925 10:12:01.332429 447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:01.349197 447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:01.351747 447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:01.351972 447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:01.352262 447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:01.352597 447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:01.352654 447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:01.352701 447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:01.352725 447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:01.358804 447 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:44179
I20260925 10:12:01.358913 575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:44179 every 8 connection(s)
I20260925 10:12:01.359243 447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:12:01.362540 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 447
I20260925 10:12:01.362635 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:12:01.362839 14501 external_mini_cluster.cc:1428] Setting key 8d2834c385a93fadaee6baf85063ae59
Sep 25 10:12:01 dist-test-slave-s6tm krb5kdc[32540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331121, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:12:01.369484 32624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.360227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:60239 (local address 127.14.41.126:45273)
0925 10:12:01.360309 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:01.360315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:01.360568 (+ 253us) server_negotiation.cc:407] Connection header received
0925 10:12:01.361112 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:01.361113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:01.361126 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:01.361184 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:01.362178 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.362788 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.364201 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.364419 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.364500 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.366723 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:01.366724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:01.366724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:01.366733 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:01.368024 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:01.368472 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:01.368473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:01.368474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:01.368539 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:01.368809 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:01.368810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:01.368811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:01.368935 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:01.369015 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:01.369302 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 10:12:01.369388 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:12:01.369688 578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.360212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:45273 (local address 127.14.41.67:60239)
0925 10:12:01.360430 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:01.360470 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:01.361001 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:01.361263 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:01.361267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:01.361876 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:01.362076 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:01.362082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.362885 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.362886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:01.364107 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:01.364110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.364507 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.364508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:01.364656 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.365144 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:01.365153 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:01.366616 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:01.368128 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:01.368136 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:01.368139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:01.368391 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:01.368616 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:01.368617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:01.368617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:01.368741 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:01.369005 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:01.369012 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:01.369162 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 10:12:01.369280 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 10:12:01.369447 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260925 10:12:01.370270 576 heartbeater.cc:347] Connected to a master server at 127.14.41.126:45273
I20260925 10:12:01.370400 576 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:01.370647 576 heartbeater.cc:511] Master 127.14.41.126:45273 requested a full tablet report, sending...
I20260925 10:12:01.371101 32585 ts_manager.cc:194] Registered new tserver with Master: c6a7a466e4be42c1b0a55229068023b5 (127.14.41.67:44179)
I20260925 10:12:01.371542 32585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:60239
I20260925 10:12:01.374713 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:12:01.382813 32624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.376314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54206 (local address 127.14.41.126:45273)
0925 10:12:01.376355 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:01.376358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:01.376371 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:12:01.376490 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:01.376491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:01.376501 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:01.376562 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:01.377160 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.377712 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.378496 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.378722 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.378807 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.380318 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:01.380319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:01.380319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:01.380326 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:01.381657 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:01.382042 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:01.382042 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:01.382043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:01.382093 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:01.382299 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:01.382300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:01.382301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:01.382468 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:01.382558 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:01.382670 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 10:12:01.382722 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:12:01.384684 32585 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:12:01.386299 32585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:12:01.393457 591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.389156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41884 (local address 127.14.41.67:44179)
0925 10:12:01.389349 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:01.389353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:01.389880 (+ 527us) server_negotiation.cc:407] Connection header received
0925 10:12:01.389926 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:01.389929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:01.389945 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:01.390013 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:01.390215 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.391015 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.392458 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.393049 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.393136 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.393199 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:01.393333 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 10:12:01.393375 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
I20260925 10:12:01.393468 587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.388907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:44179 (local address 127.0.0.1:41884)
0925 10:12:01.389769 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:01.389777 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:01.389820 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:01.390097 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:01.390097 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:01.390099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:01.390133 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:01.390134 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.391253 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.391254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:01.392351 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:01.392353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.393137 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.393138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:01.393215 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.393230 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 10:12:01.393311 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 10:12:01.393369 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":74,"thread_start_us":92,"threads_started":1}
I20260925 10:12:01.394027 589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.388902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:42591 (local address 127.0.0.1:47396)
0925 10:12:01.389382 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:01.389422 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:01.389487 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:01.389841 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:01.389844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:01.389847 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:01.389932 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:01.389934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.391312 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.391312 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:01.392831 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:01.392834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.393764 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.393765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:01.393830 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.393835 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:12:01.393884 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 10:12:01.393932 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":60,"thread_start_us":66,"threads_started":1}
I20260925 10:12:01.394037 588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.388907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:42279 (local address 127.0.0.1:58386)
0925 10:12:01.389625 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:01.389634 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:01.389693 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:01.389955 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:01.389956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:01.389958 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:01.390021 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:01.390023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.391313 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.391314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:01.392613 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:01.392616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.393777 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.393778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:01.393842 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.393847 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:12:01.393900 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 10:12:01.393943 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":52,"thread_start_us":99,"threads_started":1}
I20260925 10:12:01.394192 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.389184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58386 (local address 127.14.41.65:42279)
0925 10:12:01.389364 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:01.389368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:01.389683 (+ 315us) server_negotiation.cc:407] Connection header received
0925 10:12:01.389767 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:01.389769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:01.389790 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:01.389865 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:01.390100 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.391184 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.392706 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.393416 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.393477 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.393529 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:01.394075 (+ 546us) server_negotiation.cc:299] Negotiation successful
0925 10:12:01.394119 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":80,"threads_started":1}
I20260925 10:12:01.394381 590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.389015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47396 (local address 127.14.41.66:42591)
0925 10:12:01.389193 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:01.389198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:01.389500 (+ 302us) server_negotiation.cc:407] Connection header received
0925 10:12:01.389619 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:01.389621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:01.389639 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:01.389740 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:01.389998 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.391189 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.392920 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.393675 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.393763 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.393818 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:01.394270 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 10:12:01.394320 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20260925 10:12:01.394448 510 tablet_service.cc:1467] Processing CreateTablet for tablet 110583f66f364975bc17ceb5e851357f (DEFAULT_TABLE table=test-table [id=c7c106a5d4e44d689606eed0643f6e4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:01.394857 510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110583f66f364975bc17ceb5e851357f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:01.394863 32710 tablet_service.cc:1467] Processing CreateTablet for tablet 110583f66f364975bc17ceb5e851357f (DEFAULT_TABLE table=test-table [id=c7c106a5d4e44d689606eed0643f6e4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:01.395126 32710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110583f66f364975bc17ceb5e851357f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:01.395063 376 tablet_service.cc:1467] Processing CreateTablet for tablet 110583f66f364975bc17ceb5e851357f (DEFAULT_TABLE table=test-table [id=c7c106a5d4e44d689606eed0643f6e4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:01.395382 376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 110583f66f364975bc17ceb5e851357f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:01.397984 593 tablet_bootstrap.cc:492] T 110583f66f364975bc17ceb5e851357f P 8bd6369556e145408e4a53eab11ef14d: Bootstrap starting.
I20260925 10:12:01.398705 593 tablet_bootstrap.cc:654] T 110583f66f364975bc17ceb5e851357f P 8bd6369556e145408e4a53eab11ef14d: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:01.398730 594 tablet_bootstrap.cc:492] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5: Bootstrap starting.
I20260925 10:12:01.399369 595 tablet_bootstrap.cc:492] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12: Bootstrap starting.
I20260925 10:12:01.399379 593 log.cc:824] T 110583f66f364975bc17ceb5e851357f P 8bd6369556e145408e4a53eab11ef14d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:01.399484 594 tablet_bootstrap.cc:654] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:01.400070 595 tablet_bootstrap.cc:654] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:01.400135 594 log.cc:824] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:01.400302 593 tablet_bootstrap.cc:492] T 110583f66f364975bc17ceb5e851357f P 8bd6369556e145408e4a53eab11ef14d: No bootstrap required, opened a new log
I20260925 10:12:01.400372 593 ts_tablet_manager.cc:1397] T 110583f66f364975bc17ceb5e851357f P 8bd6369556e145408e4a53eab11ef14d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:12:01.400658 595 log.cc:824] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:01.400979 594 tablet_bootstrap.cc:492] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5: No bootstrap required, opened a new log
I20260925 10:12:01.401055 594 ts_tablet_manager.cc:1397] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:12:01.401397 595 tablet_bootstrap.cc:492] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12: No bootstrap required, opened a new log
I20260925 10:12:01.401481 595 ts_tablet_manager.cc:1397] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:12:01.402503 593 raft_consensus.cc:348] T 110583f66f364975bc17ceb5e851357f P 8bd6369556e145408e4a53eab11ef14d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a7a466e4be42c1b0a55229068023b5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44179 } } peers { permanent_uuid: "8bd6369556e145408e4a53eab11ef14d" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42279 } } peers { permanent_uuid: "3c0fd31573364accafd38b0819d11e12" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 42591 } }
I20260925 10:12:01.402706 593 raft_consensus.cc:374] T 110583f66f364975bc17ceb5e851357f P 8bd6369556e145408e4a53eab11ef14d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:01.402745 593 raft_consensus.cc:729] T 110583f66f364975bc17ceb5e851357f P 8bd6369556e145408e4a53eab11ef14d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bd6369556e145408e4a53eab11ef14d, State: Initialized, Role: FOLLOWER
I20260925 10:12:01.402894 593 consensus_queue.cc:260] T 110583f66f364975bc17ceb5e851357f P 8bd6369556e145408e4a53eab11ef14d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a7a466e4be42c1b0a55229068023b5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44179 } } peers { permanent_uuid: "8bd6369556e145408e4a53eab11ef14d" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42279 } } peers { permanent_uuid: "3c0fd31573364accafd38b0819d11e12" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 42591 } }
I20260925 10:12:01.403052 594 raft_consensus.cc:348] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a7a466e4be42c1b0a55229068023b5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44179 } } peers { permanent_uuid: "8bd6369556e145408e4a53eab11ef14d" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42279 } } peers { permanent_uuid: "3c0fd31573364accafd38b0819d11e12" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 42591 } }
I20260925 10:12:01.403216 593 ts_tablet_manager.cc:1428] T 110583f66f364975bc17ceb5e851357f P 8bd6369556e145408e4a53eab11ef14d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:12:01.403232 594 raft_consensus.cc:374] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:01.403270 594 raft_consensus.cc:729] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6a7a466e4be42c1b0a55229068023b5, State: Initialized, Role: FOLLOWER
I20260925 10:12:01.403340 308 heartbeater.cc:503] Master 127.14.41.126:45273 was elected leader, sending a full tablet report...
I20260925 10:12:01.403389 594 consensus_queue.cc:260] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a7a466e4be42c1b0a55229068023b5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44179 } } peers { permanent_uuid: "8bd6369556e145408e4a53eab11ef14d" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42279 } } peers { permanent_uuid: "3c0fd31573364accafd38b0819d11e12" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 42591 } }
I20260925 10:12:01.403286 595 raft_consensus.cc:348] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a7a466e4be42c1b0a55229068023b5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44179 } } peers { permanent_uuid: "8bd6369556e145408e4a53eab11ef14d" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42279 } } peers { permanent_uuid: "3c0fd31573364accafd38b0819d11e12" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 42591 } }
I20260925 10:12:01.403481 595 raft_consensus.cc:374] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:01.403524 595 raft_consensus.cc:729] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0fd31573364accafd38b0819d11e12, State: Initialized, Role: FOLLOWER
I20260925 10:12:01.403671 594 ts_tablet_manager.cc:1428] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 10:12:01.403671 595 consensus_queue.cc:260] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a7a466e4be42c1b0a55229068023b5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44179 } } peers { permanent_uuid: "8bd6369556e145408e4a53eab11ef14d" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42279 } } peers { permanent_uuid: "3c0fd31573364accafd38b0819d11e12" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 42591 } }
I20260925 10:12:01.403813 576 heartbeater.cc:503] Master 127.14.41.126:45273 was elected leader, sending a full tablet report...
I20260925 10:12:01.404239 442 heartbeater.cc:503] Master 127.14.41.126:45273 was elected leader, sending a full tablet report...
I20260925 10:12:01.404224 595 ts_tablet_manager.cc:1428] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260925 10:12:01.443878 443 tablet.cc:2369] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:12:01.468227 601 raft_consensus.cc:482] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:12:01.468395 601 raft_consensus.cc:504] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a7a466e4be42c1b0a55229068023b5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44179 } } peers { permanent_uuid: "8bd6369556e145408e4a53eab11ef14d" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42279 } } peers { permanent_uuid: "3c0fd31573364accafd38b0819d11e12" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 42591 } }
I20260925 10:12:01.468753 601 leader_election.cc:288] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6a7a466e4be42c1b0a55229068023b5 (127.14.41.67:44179), 8bd6369556e145408e4a53eab11ef14d (127.14.41.65:42279)
I20260925 10:12:01.472926 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.469021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:38005 (local address 127.14.41.65:42279)
0925 10:12:01.469072 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:01.469074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:01.469186 (+ 112us) server_negotiation.cc:407] Connection header received
0925 10:12:01.469215 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:01.469217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:01.469229 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:01.469316 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:01.469454 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.470199 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.471839 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.472492 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.472578 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.472588 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:01.472801 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 10:12:01.472847 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:12:01.472926 591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.469045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:57071 (local address 127.14.41.67:44179)
0925 10:12:01.469092 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:01.469094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:01.469102 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:12:01.469137 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:01.469137 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:01.469144 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:01.469229 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:01.469458 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.470234 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.471839 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.472493 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.472578 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.472588 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:01.472824 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 10:12:01.472866 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:12:01.472934 444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.468928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:44179 (local address 127.14.41.66:57071)
0925 10:12:01.468970 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:01.468984 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:01.469034 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:01.469278 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:01.469279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:01.469282 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:01.469372 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:01.469374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.470324 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.470325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:01.471717 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:01.471720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.472604 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.472605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:01.472697 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.472699 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:12:01.472764 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 10:12:01.472817 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":114}
I20260925 10:12:01.472941 602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.468931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:42279 (local address 127.14.41.66:38005)
0925 10:12:01.469109 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:01.469118 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:01.469169 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:01.469298 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:01.469298 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:01.469300 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:01.469372 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:01.469374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.470307 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.470308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:01.471717 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:01.471720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.472604 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.472605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:01.472693 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.472698 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:12:01.472785 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 10:12:01.472817 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":76,"thread_start_us":101,"threads_started":1}
I20260925 10:12:01.473344 530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110583f66f364975bc17ceb5e851357f" candidate_uuid: "3c0fd31573364accafd38b0819d11e12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6a7a466e4be42c1b0a55229068023b5" is_pre_election: true
I20260925 10:12:01.473388 32730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110583f66f364975bc17ceb5e851357f" candidate_uuid: "3c0fd31573364accafd38b0819d11e12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bd6369556e145408e4a53eab11ef14d" is_pre_election: true
I20260925 10:12:01.473524 530 raft_consensus.cc:2454] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0fd31573364accafd38b0819d11e12 in term 0.
I20260925 10:12:01.473564 32730 raft_consensus.cc:2454] T 110583f66f364975bc17ceb5e851357f P 8bd6369556e145408e4a53eab11ef14d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c0fd31573364accafd38b0819d11e12 in term 0.
I20260925 10:12:01.473773 331 leader_election.cc:302] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c0fd31573364accafd38b0819d11e12, c6a7a466e4be42c1b0a55229068023b5; no voters:
I20260925 10:12:01.473919 601 raft_consensus.cc:2781] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:12:01.473977 601 raft_consensus.cc:482] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:12:01.474011 601 raft_consensus.cc:3037] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:01.474788 601 raft_consensus.cc:504] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a7a466e4be42c1b0a55229068023b5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44179 } } peers { permanent_uuid: "8bd6369556e145408e4a53eab11ef14d" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42279 } } peers { permanent_uuid: "3c0fd31573364accafd38b0819d11e12" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 42591 } }
I20260925 10:12:01.475078 601 leader_election.cc:288] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [CANDIDATE]: Term 1 election: Requested vote from peers c6a7a466e4be42c1b0a55229068023b5 (127.14.41.67:44179), 8bd6369556e145408e4a53eab11ef14d (127.14.41.65:42279)
I20260925 10:12:01.475188 530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110583f66f364975bc17ceb5e851357f" candidate_uuid: "3c0fd31573364accafd38b0819d11e12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6a7a466e4be42c1b0a55229068023b5"
I20260925 10:12:01.475286 530 raft_consensus.cc:3037] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:01.475296 32730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "110583f66f364975bc17ceb5e851357f" candidate_uuid: "3c0fd31573364accafd38b0819d11e12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bd6369556e145408e4a53eab11ef14d"
I20260925 10:12:01.475360 32730 raft_consensus.cc:3037] T 110583f66f364975bc17ceb5e851357f P 8bd6369556e145408e4a53eab11ef14d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:01.476148 32730 raft_consensus.cc:2454] T 110583f66f364975bc17ceb5e851357f P 8bd6369556e145408e4a53eab11ef14d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0fd31573364accafd38b0819d11e12 in term 1.
I20260925 10:12:01.476153 530 raft_consensus.cc:2454] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c0fd31573364accafd38b0819d11e12 in term 1.
I20260925 10:12:01.476327 329 leader_election.cc:302] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c0fd31573364accafd38b0819d11e12, 8bd6369556e145408e4a53eab11ef14d; no voters:
I20260925 10:12:01.476432 601 raft_consensus.cc:2781] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:01.476590 601 raft_consensus.cc:686] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [term 1 LEADER]: Becoming Leader. State: Replica: 3c0fd31573364accafd38b0819d11e12, State: Running, Role: LEADER
I20260925 10:12:01.476706 601 consensus_queue.cc:237] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a7a466e4be42c1b0a55229068023b5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44179 } } peers { permanent_uuid: "8bd6369556e145408e4a53eab11ef14d" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42279 } } peers { permanent_uuid: "3c0fd31573364accafd38b0819d11e12" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 42591 } }
I20260925 10:12:01.477591 32585 catalog_manager.cc:5495] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c0fd31573364accafd38b0819d11e12 (127.14.41.66). New cstate: current_term: 1 leader_uuid: "3c0fd31573364accafd38b0819d11e12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a7a466e4be42c1b0a55229068023b5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bd6369556e145408e4a53eab11ef14d" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c0fd31573364accafd38b0819d11e12" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 42591 } health_report { overall_health: HEALTHY } } }
I20260925 10:12:01.487339 590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.484764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47406 (local address 127.14.41.66:42591)
0925 10:12:01.484837 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:01.484839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:01.484849 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:12:01.484910 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:01.484911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:01.484935 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:01.485021 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:01.485268 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.485753 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.486522 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.486736 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.486812 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.486961 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:01.487065 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:01.487179 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:01.487216 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 10:12:01.487227 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 10:12:01.489387 530 raft_consensus.cc:1261] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5 [term 1 FOLLOWER]: Refusing update from remote peer 3c0fd31573364accafd38b0819d11e12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:01.489392 32730 raft_consensus.cc:1261] T 110583f66f364975bc17ceb5e851357f P 8bd6369556e145408e4a53eab11ef14d [term 1 FOLLOWER]: Refusing update from remote peer 3c0fd31573364accafd38b0819d11e12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:01.489758 601 consensus_queue.cc:1035] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6a7a466e4be42c1b0a55229068023b5" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:01.489903 603 consensus_queue.cc:1035] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bd6369556e145408e4a53eab11ef14d" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:01.492334 608 mvcc.cc:205] Tried to move back new op lower bound from 7333196273617100800 to 7333196273569288192. Current Snapshot: MvccSnapshot[applied={T|T < 7333196273617100800}]
I20260925 10:12:01.492571 607 mvcc.cc:205] Tried to move back new op lower bound from 7333196273617100800 to 7333196273569288192. Current Snapshot: MvccSnapshot[applied={T|T < 7333196273617100800}]
I20260925 10:12:01.494958 32585 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:12:01.495052 32585 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:12:01.495878 32585 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 127ca165b0794f48a91b37b5eca6c695: Sending DeleteTablet for 3 replicas of tablet 110583f66f364975bc17ceb5e851357f
I20260925 10:12:01.496212 510 tablet_service.cc:1514] Processing DeleteTablet for tablet 110583f66f364975bc17ceb5e851357f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:01 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:41884
I20260925 10:12:01.496243 32710 tablet_service.cc:1514] Processing DeleteTablet for tablet 110583f66f364975bc17ceb5e851357f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:01 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:58386
I20260925 10:12:01.496289 376 tablet_service.cc:1514] Processing DeleteTablet for tablet 110583f66f364975bc17ceb5e851357f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:01 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:47396
I20260925 10:12:01.496485 618 tablet_replica.cc:330] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12: stopping tablet replica
I20260925 10:12:01.496569 616 tablet_replica.cc:330] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5: stopping tablet replica
I20260925 10:12:01.496587 618 raft_consensus.cc:2229] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:12:01.496647 616 raft_consensus.cc:2229] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:12:01.496713 618 raft_consensus.cc:2258] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:01.496738 616 raft_consensus.cc:2258] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:01.496780 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 32647
I20260925 10:12:01.497835 618 ts_tablet_manager.cc:1905] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:01.497835 616 ts_tablet_manager.cc:1905] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:01.499771 618 ts_tablet_manager.cc:1918] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:12:01.499846 618 log.cc:1196] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/110583f66f364975bc17ceb5e851357f
I20260925 10:12:01.500039 616 ts_tablet_manager.cc:1918] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:12:01.500154 618 ts_tablet_manager.cc:1939] T 110583f66f364975bc17ceb5e851357f P 3c0fd31573364accafd38b0819d11e12: Deleting consensus metadata
I20260925 10:12:01.500278 616 log.cc:1196] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/110583f66f364975bc17ceb5e851357f
I20260925 10:12:01.500594 616 ts_tablet_manager.cc:1939] T 110583f66f364975bc17ceb5e851357f P c6a7a466e4be42c1b0a55229068023b5: Deleting consensus metadata
I20260925 10:12:01.500594 32572 catalog_manager.cc:4841] TS 3c0fd31573364accafd38b0819d11e12 (127.14.41.66:42591): tablet 110583f66f364975bc17ceb5e851357f (table test-table [id=c7c106a5d4e44d689606eed0643f6e4f]) successfully deleted
I20260925 10:12:01.501051 32571 catalog_manager.cc:4841] TS c6a7a466e4be42c1b0a55229068023b5 (127.14.41.67:44179): tablet 110583f66f364975bc17ceb5e851357f (table test-table [id=c7c106a5d4e44d689606eed0643f6e4f]) successfully deleted
W20260925 10:12:01.504292 32569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.41.65:42279 (error 108)
I20260925 10:12:01.504813 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 313
I20260925 10:12:01.504827 587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.504689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:42279 (local address 127.0.0.1:58392)
0925 10:12:01.504756 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:01.504795 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.65:42279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 10:12:01.505040 32569 catalog_manager.cc:4584] TS 8bd6369556e145408e4a53eab11ef14d (127.14.41.65:42279): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 110583f66f364975bc17ceb5e851357f: Network error: Client connection negotiation failed: client connection to 127.14.41.65:42279: connect: Connection refused (error 111)
I20260925 10:12:01.509943 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 447
I20260925 10:12:01.514645 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 32553
2026-09-25T10:12:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1987 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T10:12:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:01Z Disabled control of system clock
I20260925 10:12:01.543329 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:46479
--webserver_interface=127.14.41.126
--webserver_port=0
--builtin_ntp_servers=127.14.41.84:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:46479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:12:01.626776 623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:01.627009 623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:01.627031 623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:01.628249 623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:01.628293 623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:01.628305 623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:01.628319 623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:01.628330 623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:01.629678 623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:46479
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:46479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:01.629895 623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:01.630142 623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:01.635159 629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:01.635154 631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:01.635149 628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:01.635180 623 server_base.cc:1034] running on GCE node
I20260925 10:12:01.635743 623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:01.636108 623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:01.637280 623 hybrid_clock.cc:648] HybridClock initialized: now 1790331121637245 us; error 47 us; skew 500 ppm
I20260925 10:12:01.638564 623 webserver.cc:460] Webserver started at http://127.14.41.126:42401/ using document root <none> and password file <none>
I20260925 10:12:01.638841 623 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:01.638919 623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:01.639084 623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:01.640260 623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "68e90b3b0e9941a3a94a2e2f9e6c2b72"
format_stamp: "Formatted at 2026-09-25 10:12:01 on dist-test-slave-s6tm"
server_key: "91fa949d62225d58e752b77388bdaa0a"
server_key_iv: "5f7146ee3f0329ced50718b51503e4cf"
server_key_version: "encryptionkey@0"
I20260925 10:12:01.640640 623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "68e90b3b0e9941a3a94a2e2f9e6c2b72"
format_stamp: "Formatted at 2026-09-25 10:12:01 on dist-test-slave-s6tm"
server_key: "91fa949d62225d58e752b77388bdaa0a"
server_key_iv: "5f7146ee3f0329ced50718b51503e4cf"
server_key_version: "encryptionkey@0"
I20260925 10:12:01.641981 623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 10:12:01.642820 637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:01.643117 623 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:01.643239 623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "68e90b3b0e9941a3a94a2e2f9e6c2b72"
format_stamp: "Formatted at 2026-09-25 10:12:01 on dist-test-slave-s6tm"
server_key: "91fa949d62225d58e752b77388bdaa0a"
server_key_iv: "5f7146ee3f0329ced50718b51503e4cf"
server_key_version: "encryptionkey@0"
I20260925 10:12:01.643332 623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:01.671527 623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:01.671916 623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:01.672214 623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:01.676777 623 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:46479
I20260925 10:12:01.676811 689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:46479 every 8 connection(s)
I20260925 10:12:01.677186 623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:01.678181 690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:01.678647 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 623
I20260925 10:12:01.678766 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:01.678998 14501 external_mini_cluster.cc:1428] Setting key bbd0beb748087772cd789d59a2978020
I20260925 10:12:01.681363 690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72: Bootstrap starting.
I20260925 10:12:01.682005 690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:01.682329 690 log.cc:824] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:01.683097 690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72: No bootstrap required, opened a new log
I20260925 10:12:01.684914 693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.679574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52080 (local address 127.14.41.126:46479)
0925 10:12:01.679798 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:01.679805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:01.679830 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:12:01.680392 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:01.680400 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:01.680466 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:01.680584 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:01.680858 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.681874 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.682898 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.683383 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.683476 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.684260 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:01.684264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:01.684264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:01.684424 (+ 160us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:01.684456 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:01.684467 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:01.684516 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:01.684610 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 10:12:01.684672 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":115,"threads_started":1}
I20260925 10:12:01.685535 690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68e90b3b0e9941a3a94a2e2f9e6c2b72" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46479 } }
I20260925 10:12:01.685698 690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:01.685751 690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68e90b3b0e9941a3a94a2e2f9e6c2b72, State: Initialized, Role: FOLLOWER
I20260925 10:12:01.685875 690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68e90b3b0e9941a3a94a2e2f9e6c2b72" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46479 } }
I20260925 10:12:01.685958 690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:01.686002 690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:01.686055 690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:01.686617 690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68e90b3b0e9941a3a94a2e2f9e6c2b72" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46479 } }
I20260925 10:12:01.686739 690 leader_election.cc:302] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68e90b3b0e9941a3a94a2e2f9e6c2b72; no voters:
I20260925 10:12:01.686930 690 leader_election.cc:288] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:01.687000 695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:01.687227 690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:01.687250 695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72 [term 1 LEADER]: Becoming Leader. State: Replica: 68e90b3b0e9941a3a94a2e2f9e6c2b72, State: Running, Role: LEADER
I20260925 10:12:01.687363 695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68e90b3b0e9941a3a94a2e2f9e6c2b72" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46479 } }
I20260925 10:12:01.687763 696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68e90b3b0e9941a3a94a2e2f9e6c2b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68e90b3b0e9941a3a94a2e2f9e6c2b72" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46479 } } }
I20260925 10:12:01.687767 697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68e90b3b0e9941a3a94a2e2f9e6c2b72. Latest consensus state: current_term: 1 leader_uuid: "68e90b3b0e9941a3a94a2e2f9e6c2b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68e90b3b0e9941a3a94a2e2f9e6c2b72" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 46479 } } }
I20260925 10:12:01.687860 696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:01.687863 697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72 [sys.catalog]: This master's current role is: LEADER
W20260925 10:12:01.689235 711 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:12:01.689297 711 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:12:01.689388 704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:01.689581 704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:01.691191 704 catalog_manager.cc:1347] Generated new cluster ID: c83f893919a04d65a9cbd25ca87f4d63
I20260925 10:12:01.691247 704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:01.707803 704 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:01.708452 704 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:01.710515 704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72: Generated new TSK 0
I20260925 10:12:01.710706 704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:01.719686 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46479
--builtin_ntp_servers=127.14.41.84:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:12:01.815573 714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:01.815928 714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:01.815994 714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:01.817950 714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:01.818013 714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:01.818055 714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:12:01.820039 714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:01.820241 714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:01.820545 714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:01.825929 720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:01.825970 714 server_base.cc:1034] running on GCE node
W20260925 10:12:01.825990 719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:01.826301 722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:01.826546 714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:01.826812 714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:01.828035 714 hybrid_clock.cc:648] HybridClock initialized: now 1790331121827996 us; error 55 us; skew 500 ppm
I20260925 10:12:01.829480 714 webserver.cc:460] Webserver started at http://127.14.41.65:44007/ using document root <none> and password file <none>
I20260925 10:12:01.829752 714 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:01.829823 714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:01.829969 714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:01.831105 714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "fc9c259b4f75401181588f4bd8951aeb"
format_stamp: "Formatted at 2026-09-25 10:12:01 on dist-test-slave-s6tm"
server_key: "24670a536a243907c10ff5e2a395ed2e"
server_key_iv: "023ed7b4c69fc353057b5ed66cfa2b95"
server_key_version: "encryptionkey@0"
I20260925 10:12:01.831539 714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "fc9c259b4f75401181588f4bd8951aeb"
format_stamp: "Formatted at 2026-09-25 10:12:01 on dist-test-slave-s6tm"
server_key: "24670a536a243907c10ff5e2a395ed2e"
server_key_iv: "023ed7b4c69fc353057b5ed66cfa2b95"
server_key_version: "encryptionkey@0"
I20260925 10:12:01.833077 714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:12:01.833952 728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:01.834142 714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:12:01.834250 714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "fc9c259b4f75401181588f4bd8951aeb"
format_stamp: "Formatted at 2026-09-25 10:12:01 on dist-test-slave-s6tm"
server_key: "24670a536a243907c10ff5e2a395ed2e"
server_key_iv: "023ed7b4c69fc353057b5ed66cfa2b95"
server_key_version: "encryptionkey@0"
I20260925 10:12:01.834358 714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:01.863372 714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:01.863732 714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:01.863909 714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:01.864229 714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:01.864590 714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:01.864648 714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:01.864693 714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:01.864720 714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:01.870594 714 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:42629
I20260925 10:12:01.870677 841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:42629 every 8 connection(s)
I20260925 10:12:01.871047 714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:12:01.876402 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 714
I20260925 10:12:01.876534 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:12:01.876554 693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.871887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:42131 (local address 127.14.41.126:46479)
0925 10:12:01.872076 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:01.872078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:01.872313 (+ 235us) server_negotiation.cc:407] Connection header received
0925 10:12:01.872802 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:01.872803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:01.872819 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:01.872888 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:01.873550 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.874049 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.875046 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.875223 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.875274 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.876221 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:01.876224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:01.876224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:01.876231 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:01.876249 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:01.876254 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:01.876336 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:01.876422 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 10:12:01.876455 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260925 10:12:01.876799 14501 external_mini_cluster.cc:1428] Setting key 0e4d2079400e132deb25dfc889bfc704
I20260925 10:12:01.876811 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.871861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46479 (local address 127.14.41.65:42131)
0925 10:12:01.872130 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:01.872176 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:01.872701 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:01.872946 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:01.872949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:01.872958 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:01.873473 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:01.873478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.874140 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.874140 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:01.874966 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:01.874969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:01.875299 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:01.875300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:01.875415 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:01.876062 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:12:01.876063 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:01.876094 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:12:01.876118 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:01.876341 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:01.876345 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:01.876348 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:12:01.876402 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 10:12:01.876516 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":148,"threads_started":1}
I20260925 10:12:01.877454 842 heartbeater.cc:347] Connected to a master server at 127.14.41.126:46479
I20260925 10:12:01.877564 842 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:01.877794 842 heartbeater.cc:511] Master 127.14.41.126:46479 requested a full tablet report, sending...
I20260925 10:12:01.878103 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46479
--builtin_ntp_servers=127.14.41.84:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:12:01.878399 653 ts_manager.cc:194] Registered new tserver with Master: fc9c259b4f75401181588f4bd8951aeb (127.14.41.65:42629)
I20260925 10:12:01.879308 653 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.41.65:42131
W20260925 10:12:01.962455 845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:01.962708 845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:01.962747 845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:01.963986 845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:01.964056 845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:01.964098 845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:12:01.965502 845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:01.965793 845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:01.966111 845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:01.971555 850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:01.971555 851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:01.971630 853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:01.971843 845 server_base.cc:1034] running on GCE node
I20260925 10:12:01.972150 845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:01.972420 845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:01.973585 845 hybrid_clock.cc:648] HybridClock initialized: now 1790331121973558 us; error 40 us; skew 500 ppm
I20260925 10:12:01.974937 845 webserver.cc:460] Webserver started at http://127.14.41.66:37245/ using document root <none> and password file <none>
I20260925 10:12:01.975237 845 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:01.975317 845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:01.975481 845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:01.976673 845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "55ca665270264176a2f66fba0b402682"
format_stamp: "Formatted at 2026-09-25 10:12:01 on dist-test-slave-s6tm"
server_key: "fe13b2cdc0cb64aa80c29999a0073921"
server_key_iv: "8f3ae75e118590f970d31500907c9476"
server_key_version: "encryptionkey@0"
I20260925 10:12:01.977062 845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "55ca665270264176a2f66fba0b402682"
format_stamp: "Formatted at 2026-09-25 10:12:01 on dist-test-slave-s6tm"
server_key: "fe13b2cdc0cb64aa80c29999a0073921"
server_key_iv: "8f3ae75e118590f970d31500907c9476"
server_key_version: "encryptionkey@0"
I20260925 10:12:01.978514 845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 10:12:01.979373 859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:01.979688 845 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:01.979806 845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "55ca665270264176a2f66fba0b402682"
format_stamp: "Formatted at 2026-09-25 10:12:01 on dist-test-slave-s6tm"
server_key: "fe13b2cdc0cb64aa80c29999a0073921"
server_key_iv: "8f3ae75e118590f970d31500907c9476"
server_key_version: "encryptionkey@0"
I20260925 10:12:01.979889 845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:01.990655 845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:01.991026 845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:01.991191 845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:01.991431 845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:01.991793 845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:01.991848 845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:01.991894 845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:01.991918 845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:01.998396 845 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:33999
I20260925 10:12:01.998447 972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:33999 every 8 connection(s)
I20260925 10:12:01.998808 845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:12:02.003579 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 845
I20260925 10:12:02.003734 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:12:02.004019 14501 external_mini_cluster.cc:1428] Setting key d43998e7eae14e80aae8b3b38a2d130b
I20260925 10:12:02.004611 693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.999608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:48607 (local address 127.14.41.126:46479)
0925 10:12:01.999681 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:01.999683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:01.999977 (+ 294us) server_negotiation.cc:407] Connection header received
0925 10:12:02.000546 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:02.000548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:02.000558 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:02.000613 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:02.001315 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.001813 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.002875 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.003072 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.003132 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:02.004252 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:02.004253 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:02.004254 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:02.004260 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:02.004282 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:02.004286 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:02.004360 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:02.004496 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 10:12:02.004527 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:12:02.004844 975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:01.999612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46479 (local address 127.14.41.66:48607)
0925 10:12:01.999799 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:01.999837 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:02.000447 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:02.000673 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:02.000677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:02.000686 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:02.001220 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:02.001224 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.001905 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.001906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:02.002775 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:02.002778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.003155 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.003156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:02.003277 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:02.004047 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:12:02.004048 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:02.004094 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:12:02.004127 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:02.004390 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:02.004398 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:02.004402 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:12:02.004489 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 10:12:02.004581 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260925 10:12:02.005349 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46479
--builtin_ntp_servers=127.14.41.84:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:12:02.005396 973 heartbeater.cc:347] Connected to a master server at 127.14.41.126:46479
I20260925 10:12:02.005493 973 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:02.005688 973 heartbeater.cc:511] Master 127.14.41.126:46479 requested a full tablet report, sending...
I20260925 10:12:02.006057 653 ts_manager.cc:194] Registered new tserver with Master: 55ca665270264176a2f66fba0b402682 (127.14.41.66:33999)
I20260925 10:12:02.006471 653 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.41.66:48607
W20260925 10:12:02.089696 976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:02.089910 976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:02.089931 976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:02.091136 976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:02.091181 976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:02.091207 976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:12:02.093180 976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:46479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:02.093453 976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:02.093777 976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:02.099144 984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:02.099184 981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:02.099119 982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:02.099342 976 server_base.cc:1034] running on GCE node
I20260925 10:12:02.099570 976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:02.099835 976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:02.100996 976 hybrid_clock.cc:648] HybridClock initialized: now 1790331122100976 us; error 45 us; skew 500 ppm
I20260925 10:12:02.102288 976 webserver.cc:460] Webserver started at http://127.14.41.67:42287/ using document root <none> and password file <none>
I20260925 10:12:02.102560 976 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:02.102631 976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:02.102799 976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:02.103894 976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "60b4833a6ed14b0b9aaf9de71f1cc698"
format_stamp: "Formatted at 2026-09-25 10:12:02 on dist-test-slave-s6tm"
server_key: "814ae1d67bacbfa4f7a2469e64ff5b26"
server_key_iv: "08b9dea7f67e3766508faec2fe67c0c8"
server_key_version: "encryptionkey@0"
I20260925 10:12:02.104323 976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "60b4833a6ed14b0b9aaf9de71f1cc698"
format_stamp: "Formatted at 2026-09-25 10:12:02 on dist-test-slave-s6tm"
server_key: "814ae1d67bacbfa4f7a2469e64ff5b26"
server_key_iv: "08b9dea7f67e3766508faec2fe67c0c8"
server_key_version: "encryptionkey@0"
I20260925 10:12:02.105679 976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:02.106426 990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:02.106598 976 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:12:02.106701 976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "60b4833a6ed14b0b9aaf9de71f1cc698"
format_stamp: "Formatted at 2026-09-25 10:12:02 on dist-test-slave-s6tm"
server_key: "814ae1d67bacbfa4f7a2469e64ff5b26"
server_key_iv: "08b9dea7f67e3766508faec2fe67c0c8"
server_key_version: "encryptionkey@0"
I20260925 10:12:02.106799 976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:02.130182 976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:02.130529 976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:02.130699 976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:02.130977 976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:02.131345 976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:02.131403 976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:02.131440 976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:02.131487 976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:02.137351 976 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:44193
I20260925 10:12:02.137501 1103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:44193 every 8 connection(s)
I20260925 10:12:02.137831 976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:12:02.140991 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 976
I20260925 10:12:02.141098 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:12:02.141317 14501 external_mini_cluster.cc:1428] Setting key ab60cbfc5186958edd886cb44ed5710c
I20260925 10:12:02.143460 693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:02.138617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:38217 (local address 127.14.41.126:46479)
0925 10:12:02.138685 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:02.138687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:02.138902 (+ 215us) server_negotiation.cc:407] Connection header received
0925 10:12:02.139376 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:02.139377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:02.139387 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:02.139451 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:02.140164 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.140671 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.141998 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.142308 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.142391 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:02.143220 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:02.143221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:02.143222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:02.143226 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:02.143240 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:02.143244 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:02.143311 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:02.143377 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 10:12:02.143401 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:12:02.143693 1106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:02.138571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:46479 (local address 127.14.41.67:38217)
0925 10:12:02.138764 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:02.138808 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:02.139289 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:02.139519 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:02.139522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:02.139531 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:02.140086 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:02.140092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.140796 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.140797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:02.141932 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:02.141935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.142428 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.142430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:02.142552 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:02.143083 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:12:02.143084 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:02.143114 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:12:02.143137 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:02.143319 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:02.143323 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:02.143326 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:12:02.143380 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 10:12:02.143465 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260925 10:12:02.144236 1104 heartbeater.cc:347] Connected to a master server at 127.14.41.126:46479
I20260925 10:12:02.144332 1104 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:02.144485 1104 heartbeater.cc:511] Master 127.14.41.126:46479 requested a full tablet report, sending...
I20260925 10:12:02.144840 653 ts_manager.cc:194] Registered new tserver with Master: 60b4833a6ed14b0b9aaf9de71f1cc698 (127.14.41.67:44193)
I20260925 10:12:02.145210 653 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.41.67:38217
I20260925 10:12:02.153471 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:12:02.158339 693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:02.155016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45052 (local address 127.14.41.126:46479)
0925 10:12:02.155099 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:02.155101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:02.155209 (+ 108us) server_negotiation.cc:407] Connection header received
0925 10:12:02.155331 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:02.155332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:02.155343 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:02.155404 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:02.155594 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.156180 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.156952 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.157148 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.157245 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:02.158023 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:02.158024 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:02.158025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:02.158030 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:02.158044 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:02.158048 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:02.158167 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:02.158232 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 10:12:02.158246 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:12:02.160259 653 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:12:02.161870 653 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:12:02.168571 1115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:02.164300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60262 (local address 127.14.41.65:42629)
0925 10:12:02.164514 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:02.164517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:02.164829 (+ 312us) server_negotiation.cc:407] Connection header received
0925 10:12:02.164915 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:02.164917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:02.164933 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:02.165012 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:02.165244 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.166349 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.167469 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.168141 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.168230 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:02.168330 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:02.168432 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 10:12:02.168470 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
I20260925 10:12:02.168558 1116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:02.164103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:42629 (local address 127.0.0.1:60262)
0925 10:12:02.164759 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:02.164770 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:02.164827 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:02.165098 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:02.165100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:02.165102 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:02.165158 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:02.165160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.166433 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.166434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:02.167372 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:02.167374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.168241 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.168242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:02.168321 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:02.168336 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 10:12:02.168415 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 10:12:02.168471 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":60,"threads_started":1}
I20260925 10:12:02.169039 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:02.164348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:44193 (local address 127.0.0.1:43298)
0925 10:12:02.164911 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:02.164921 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:02.164964 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:02.165211 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:02.165212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:02.165214 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:02.165251 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:02.165253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.166112 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.166113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:02.167394 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:02.167397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.168472 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.168473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:02.168533 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:02.168538 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:12:02.168586 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 10:12:02.168974 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":70,"threads_started":1}
I20260925 10:12:02.169050 1114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:02.164194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57330 (local address 127.14.41.66:33999)
0925 10:12:02.164400 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:02.164405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:02.164516 (+ 111us) server_negotiation.cc:407] Connection header received
0925 10:12:02.164606 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:02.164608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:02.164621 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:02.164727 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:02.164921 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.166301 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.167595 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.168513 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.168576 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:02.168656 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:02.168921 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 10:12:02.168964 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":90,"threads_started":1}
I20260925 10:12:02.169274 1113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:02.164101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:33999 (local address 127.0.0.1:57330)
0925 10:12:02.164391 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:02.164434 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:02.164517 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:02.164753 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:02.164756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:02.164759 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:02.164845 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:02.164847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.166396 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.166397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:02.167510 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:02.167512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.168695 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.168696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:02.168768 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:02.168771 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:12:02.168816 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 10:12:02.169204 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":45,"thread_start_us":70,"threads_started":1}
I20260925 10:12:02.169425 776 tablet_service.cc:1467] Processing CreateTablet for tablet bd2196125e534046bcd442b264fc255d (DEFAULT_TABLE table=test-table [id=156c84194e184c59af85a7a757ccb407]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:02.169785 1117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:02.164554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.14.41.67:44193)
0925 10:12:02.164753 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:02.164757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:02.164967 (+ 210us) server_negotiation.cc:407] Connection header received
0925 10:12:02.165115 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:02.165118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:02.165134 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:02.165204 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:02.165303 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.166021 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.167502 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.168282 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.168349 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:02.168436 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:02.169589 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0925 10:12:02.169681 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":97,"threads_started":1}
I20260925 10:12:02.169854 776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd2196125e534046bcd442b264fc255d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:02.170423 1038 tablet_service.cc:1467] Processing CreateTablet for tablet bd2196125e534046bcd442b264fc255d (DEFAULT_TABLE table=test-table [id=156c84194e184c59af85a7a757ccb407]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:02.170724 1038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd2196125e534046bcd442b264fc255d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:02.172412 907 tablet_service.cc:1467] Processing CreateTablet for tablet bd2196125e534046bcd442b264fc255d (DEFAULT_TABLE table=test-table [id=156c84194e184c59af85a7a757ccb407]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:02.172752 907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd2196125e534046bcd442b264fc255d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:02.173053 1118 tablet_bootstrap.cc:492] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb: Bootstrap starting.
I20260925 10:12:02.173470 1119 tablet_bootstrap.cc:492] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698: Bootstrap starting.
I20260925 10:12:02.173763 1118 tablet_bootstrap.cc:654] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:02.174135 1119 tablet_bootstrap.cc:654] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:02.174113 1118 log.cc:824] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:02.174503 1119 log.cc:824] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:02.174984 1118 tablet_bootstrap.cc:492] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb: No bootstrap required, opened a new log
I20260925 10:12:02.175082 1118 ts_tablet_manager.cc:1397] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:12:02.175269 1119 tablet_bootstrap.cc:492] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698: No bootstrap required, opened a new log
I20260925 10:12:02.175359 1119 ts_tablet_manager.cc:1397] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:12:02.176292 1122 tablet_bootstrap.cc:492] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682: Bootstrap starting.
I20260925 10:12:02.177013 1122 tablet_bootstrap.cc:654] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:02.177387 1122 log.cc:824] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:02.177383 1118 raft_consensus.cc:348] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ca665270264176a2f66fba0b402682" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 33999 } } peers { permanent_uuid: "fc9c259b4f75401181588f4bd8951aeb" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42629 } } peers { permanent_uuid: "60b4833a6ed14b0b9aaf9de71f1cc698" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44193 } }
I20260925 10:12:02.177549 1118 raft_consensus.cc:374] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:02.177595 1118 raft_consensus.cc:729] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc9c259b4f75401181588f4bd8951aeb, State: Initialized, Role: FOLLOWER
I20260925 10:12:02.177706 1118 consensus_queue.cc:260] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ca665270264176a2f66fba0b402682" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 33999 } } peers { permanent_uuid: "fc9c259b4f75401181588f4bd8951aeb" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42629 } } peers { permanent_uuid: "60b4833a6ed14b0b9aaf9de71f1cc698" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44193 } }
I20260925 10:12:02.177994 1118 ts_tablet_manager.cc:1428] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 10:12:02.178051 842 heartbeater.cc:503] Master 127.14.41.126:46479 was elected leader, sending a full tablet report...
I20260925 10:12:02.178385 1119 raft_consensus.cc:348] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ca665270264176a2f66fba0b402682" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 33999 } } peers { permanent_uuid: "fc9c259b4f75401181588f4bd8951aeb" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42629 } } peers { permanent_uuid: "60b4833a6ed14b0b9aaf9de71f1cc698" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44193 } }
I20260925 10:12:02.178550 1119 raft_consensus.cc:374] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:02.178583 1119 raft_consensus.cc:729] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b4833a6ed14b0b9aaf9de71f1cc698, State: Initialized, Role: FOLLOWER
I20260925 10:12:02.178699 1119 consensus_queue.cc:260] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ca665270264176a2f66fba0b402682" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 33999 } } peers { permanent_uuid: "fc9c259b4f75401181588f4bd8951aeb" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42629 } } peers { permanent_uuid: "60b4833a6ed14b0b9aaf9de71f1cc698" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44193 } }
I20260925 10:12:02.178956 1119 ts_tablet_manager.cc:1428] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 10:12:02.178980 1104 heartbeater.cc:503] Master 127.14.41.126:46479 was elected leader, sending a full tablet report...
I20260925 10:12:02.179688 1122 tablet_bootstrap.cc:492] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682: No bootstrap required, opened a new log
I20260925 10:12:02.179771 1122 ts_tablet_manager.cc:1397] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 10:12:02.181562 1122 raft_consensus.cc:348] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ca665270264176a2f66fba0b402682" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 33999 } } peers { permanent_uuid: "fc9c259b4f75401181588f4bd8951aeb" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42629 } } peers { permanent_uuid: "60b4833a6ed14b0b9aaf9de71f1cc698" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44193 } }
I20260925 10:12:02.181715 1122 raft_consensus.cc:374] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:02.181775 1122 raft_consensus.cc:729] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ca665270264176a2f66fba0b402682, State: Initialized, Role: FOLLOWER
I20260925 10:12:02.181890 1122 consensus_queue.cc:260] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ca665270264176a2f66fba0b402682" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 33999 } } peers { permanent_uuid: "fc9c259b4f75401181588f4bd8951aeb" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42629 } } peers { permanent_uuid: "60b4833a6ed14b0b9aaf9de71f1cc698" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44193 } }
I20260925 10:12:02.182164 1122 ts_tablet_manager.cc:1428] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 10:12:02.182222 973 heartbeater.cc:503] Master 127.14.41.126:46479 was elected leader, sending a full tablet report...
I20260925 10:12:02.229077 1124 raft_consensus.cc:482] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:12:02.229259 1124 raft_consensus.cc:504] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ca665270264176a2f66fba0b402682" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 33999 } } peers { permanent_uuid: "fc9c259b4f75401181588f4bd8951aeb" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42629 } } peers { permanent_uuid: "60b4833a6ed14b0b9aaf9de71f1cc698" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44193 } }
I20260925 10:12:02.229647 1124 leader_election.cc:288] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55ca665270264176a2f66fba0b402682 (127.14.41.66:33999), 60b4833a6ed14b0b9aaf9de71f1cc698 (127.14.41.67:44193)
I20260925 10:12:02.232806 1114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:02.229942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:48025 (local address 127.14.41.66:33999)
0925 10:12:02.230006 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:02.230009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:02.230019 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:12:02.230052 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:02.230053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:02.230066 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:02.230139 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:02.230357 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.230950 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.231982 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.232439 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.232499 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:02.232506 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:02.232706 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 10:12:02.232752 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 10:12:02.232810 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:02.229856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:33999 (local address 127.14.41.65:48025)
0925 10:12:02.229920 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:02.229933 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:02.229979 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:02.230160 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:02.230161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:02.230164 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:02.230263 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:02.230265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.231036 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.231037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:02.231878 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:02.231881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.232528 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.232529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:02.232617 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:02.232621 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:12:02.232686 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 10:12:02.232729 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 10:12:02.233294 1127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:02.229985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:44193 (local address 127.14.41.65:35379)
0925 10:12:02.230157 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:02.230167 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:02.230217 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:02.230412 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:02.230412 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:02.230415 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:02.230482 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:02.230484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.231147 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.231148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:02.232135 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:02.232138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.233001 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.233003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:02.233109 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:02.233113 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:12:02.233175 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 10:12:02.233210 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":102,"threads_started":1}
I20260925 10:12:02.233222 927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2196125e534046bcd442b264fc255d" candidate_uuid: "fc9c259b4f75401181588f4bd8951aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ca665270264176a2f66fba0b402682" is_pre_election: true
I20260925 10:12:02.233312 1117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:02.230131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:35379 (local address 127.14.41.67:44193)
0925 10:12:02.230190 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:02.230192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:02.230206 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:12:02.230321 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:02.230322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:02.230338 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:02.230396 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:02.230570 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.231061 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.232265 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.232793 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.232879 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:02.232888 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:02.233199 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 10:12:02.233251 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 10:12:02.233403 927 raft_consensus.cc:2454] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc9c259b4f75401181588f4bd8951aeb in term 0.
I20260925 10:12:02.233639 732 leader_election.cc:302] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55ca665270264176a2f66fba0b402682, fc9c259b4f75401181588f4bd8951aeb; no voters:
I20260925 10:12:02.233695 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2196125e534046bcd442b264fc255d" candidate_uuid: "fc9c259b4f75401181588f4bd8951aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b4833a6ed14b0b9aaf9de71f1cc698" is_pre_election: true
I20260925 10:12:02.233820 1124 raft_consensus.cc:2781] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:12:02.233856 1058 raft_consensus.cc:2454] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc9c259b4f75401181588f4bd8951aeb in term 0.
I20260925 10:12:02.233880 1124 raft_consensus.cc:482] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:12:02.233915 1124 raft_consensus.cc:3037] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:02.234649 1124 raft_consensus.cc:504] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ca665270264176a2f66fba0b402682" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 33999 } } peers { permanent_uuid: "fc9c259b4f75401181588f4bd8951aeb" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42629 } } peers { permanent_uuid: "60b4833a6ed14b0b9aaf9de71f1cc698" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44193 } }
I20260925 10:12:02.234756 1124 leader_election.cc:288] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [CANDIDATE]: Term 1 election: Requested vote from peers 55ca665270264176a2f66fba0b402682 (127.14.41.66:33999), 60b4833a6ed14b0b9aaf9de71f1cc698 (127.14.41.67:44193)
I20260925 10:12:02.234913 927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2196125e534046bcd442b264fc255d" candidate_uuid: "fc9c259b4f75401181588f4bd8951aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ca665270264176a2f66fba0b402682"
I20260925 10:12:02.234961 1058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2196125e534046bcd442b264fc255d" candidate_uuid: "fc9c259b4f75401181588f4bd8951aeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b4833a6ed14b0b9aaf9de71f1cc698"
I20260925 10:12:02.235023 1058 raft_consensus.cc:3037] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:02.235018 927 raft_consensus.cc:3037] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:02.235993 1058 raft_consensus.cc:2454] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc9c259b4f75401181588f4bd8951aeb in term 1.
I20260925 10:12:02.235993 927 raft_consensus.cc:2454] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc9c259b4f75401181588f4bd8951aeb in term 1.
I20260925 10:12:02.236174 732 leader_election.cc:302] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55ca665270264176a2f66fba0b402682, fc9c259b4f75401181588f4bd8951aeb; no voters:
I20260925 10:12:02.236328 1124 raft_consensus.cc:2781] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:02.236511 1124 raft_consensus.cc:686] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [term 1 LEADER]: Becoming Leader. State: Replica: fc9c259b4f75401181588f4bd8951aeb, State: Running, Role: LEADER
I20260925 10:12:02.236619 1124 consensus_queue.cc:237] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ca665270264176a2f66fba0b402682" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 33999 } } peers { permanent_uuid: "fc9c259b4f75401181588f4bd8951aeb" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42629 } } peers { permanent_uuid: "60b4833a6ed14b0b9aaf9de71f1cc698" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44193 } }
I20260925 10:12:02.237617 653 catalog_manager.cc:5495] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to fc9c259b4f75401181588f4bd8951aeb (127.14.41.65). New cstate: current_term: 1 leader_uuid: "fc9c259b4f75401181588f4bd8951aeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ca665270264176a2f66fba0b402682" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 33999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc9c259b4f75401181588f4bd8951aeb" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 42629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60b4833a6ed14b0b9aaf9de71f1cc698" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44193 } health_report { overall_health: UNKNOWN } } }
I20260925 10:12:02.245949 1115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:02.243585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60272 (local address 127.14.41.65:42629)
0925 10:12:02.243646 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:02.243648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:02.243658 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:12:02.243691 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:02.243692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:02.243709 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:02.243770 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:02.243901 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.244435 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.245164 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:02.245378 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:02.245432 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:02.245619 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:02.245707 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:02.245782 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:02.245828 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 10:12:02.245838 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 10:12:02.247895 1058 raft_consensus.cc:1261] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698 [term 1 FOLLOWER]: Refusing update from remote peer fc9c259b4f75401181588f4bd8951aeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:02.247984 927 raft_consensus.cc:1261] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682 [term 1 FOLLOWER]: Refusing update from remote peer fc9c259b4f75401181588f4bd8951aeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:02.248343 1124 consensus_queue.cc:1035] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b4833a6ed14b0b9aaf9de71f1cc698" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 44193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:02.248517 1128 consensus_queue.cc:1035] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "55ca665270264176a2f66fba0b402682" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 33999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 10:12:02.249950 974 tablet.cc:2369] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:12:02.251302 1133 mvcc.cc:205] Tried to move back new op lower bound from 7333196276724531200 to 7333196276681969664. Current Snapshot: MvccSnapshot[applied={T|T < 7333196276724531200}]
I20260925 10:12:02.253398 653 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:12:02.253502 653 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:12:02.254268 653 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68e90b3b0e9941a3a94a2e2f9e6c2b72: Sending DeleteTablet for 3 replicas of tablet bd2196125e534046bcd442b264fc255d
I20260925 10:12:02.254603 907 tablet_service.cc:1514] Processing DeleteTablet for tablet bd2196125e534046bcd442b264fc255d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:02 UTC) from {username='slave'} at 127.0.0.1:57330
I20260925 10:12:02.254592 776 tablet_service.cc:1514] Processing DeleteTablet for tablet bd2196125e534046bcd442b264fc255d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:02 UTC) from {username='slave'} at 127.0.0.1:60262
I20260925 10:12:02.254707 1038 tablet_service.cc:1514] Processing DeleteTablet for tablet bd2196125e534046bcd442b264fc255d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:02 UTC) from {username='slave'} at 127.0.0.1:43298
I20260925 10:12:02.255041 1142 tablet_replica.cc:330] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb: stopping tablet replica
I20260925 10:12:02.255075 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 714
I20260925 10:12:02.255143 1142 raft_consensus.cc:2229] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:12:02.255244 1142 raft_consensus.cc:2258] T bd2196125e534046bcd442b264fc255d P fc9c259b4f75401181588f4bd8951aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:02.255386 1141 tablet_replica.cc:330] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682: stopping tablet replica
I20260925 10:12:02.255466 1141 raft_consensus.cc:2229] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:12:02.255548 1141 raft_consensus.cc:2258] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:02.255705 1143 tablet_replica.cc:330] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698: stopping tablet replica
I20260925 10:12:02.255810 1143 raft_consensus.cc:2229] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:12:02.255919 1143 raft_consensus.cc:2258] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:02.256039 1141 ts_tablet_manager.cc:1905] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:02.256580 1143 ts_tablet_manager.cc:1905] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:02.257835 1141 ts_tablet_manager.cc:1918] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:12:02.257917 1141 log.cc:1196] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/bd2196125e534046bcd442b264fc255d
I20260925 10:12:02.258175 1141 ts_tablet_manager.cc:1939] T bd2196125e534046bcd442b264fc255d P 55ca665270264176a2f66fba0b402682: Deleting consensus metadata
I20260925 10:12:02.258258 1143 ts_tablet_manager.cc:1918] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:12:02.258333 1143 log.cc:1196] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/bd2196125e534046bcd442b264fc255d
I20260925 10:12:02.258577 1143 ts_tablet_manager.cc:1939] T bd2196125e534046bcd442b264fc255d P 60b4833a6ed14b0b9aaf9de71f1cc698: Deleting consensus metadata
I20260925 10:12:02.258711 641 catalog_manager.cc:4841] TS 55ca665270264176a2f66fba0b402682 (127.14.41.66:33999): tablet bd2196125e534046bcd442b264fc255d (table test-table [id=156c84194e184c59af85a7a757ccb407]) successfully deleted
I20260925 10:12:02.258932 641 catalog_manager.cc:4841] TS 60b4833a6ed14b0b9aaf9de71f1cc698 (127.14.41.67:44193): tablet bd2196125e534046bcd442b264fc255d (table test-table [id=156c84194e184c59af85a7a757ccb407]) successfully deleted
W20260925 10:12:02.261754 638 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.41.65:42629 (error 108)
I20260925 10:12:02.261886 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 845
I20260925 10:12:02.263214 1113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:02.263038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:42629 (local address 127.0.0.1:60280)
0925 10:12:02.263114 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:02.263167 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.65:42629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 10:12:02.267489 638 catalog_manager.cc:4584] TS fc9c259b4f75401181588f4bd8951aeb (127.14.41.65:42629): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd2196125e534046bcd442b264fc255d: Network error: Client connection negotiation failed: client connection to 127.14.41.65:42629: connect: Connection refused (error 111)
I20260925 10:12:02.267647 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 976
I20260925 10:12:02.273097 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 623
2026-09-25T10:12:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (757 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:02 dist-test-slave-s6tm krb5kdc[1147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:02 dist-test-slave-s6tm krb5kdc[1147](info): set up 2 sockets
Sep 25 10:12:02 dist-test-slave-s6tm krb5kdc[1147](info): commencing operation
krb5kdc: starting...
W20260925 10:12:03.353367 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:03 dist-test-slave-s6tm krb5kdc[1147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:12:03.480122 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:39379
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:39379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:03.564654 1160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:03.564931 1160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:03.564973 1160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:03.566202 1160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:03.566288 1160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:03.566313 1160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:03.566349 1160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:03.566378 1160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:03.568007 1160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:39379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:39379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:03.568328 1160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:03.568600 1160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:03.573880 1166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:03.573889 1165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:03.573895 1168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:03.573915 1160 server_base.cc:1034] running on GCE node
I20260925 10:12:03.574517 1160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:03.574848 1160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:03.576043 1160 hybrid_clock.cc:648] HybridClock initialized: now 1790331123576006 us; error 60 us; skew 500 ppm
Sep 25 10:12:03 dist-test-slave-s6tm krb5kdc[1147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331123, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:03.578754 1160 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:12:03.579294 1160 webserver.cc:460] Webserver started at http://127.14.41.126:43947/ using document root <none> and password file <none>
I20260925 10:12:03.579551 1160 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:03.579623 1160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:03.579784 1160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:03.580983 1160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "1ea9e59dbe9041c2844385a852e01909"
format_stamp: "Formatted at 2026-09-25 10:12:03 on dist-test-slave-s6tm"
server_key: "ecf85e9ada4e3cf36f2c5ac4695fd440"
server_key_iv: "26a429a418a6b9f9ef1d5a7213b18fe2"
server_key_version: "encryptionkey@0"
I20260925 10:12:03.581362 1160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "1ea9e59dbe9041c2844385a852e01909"
format_stamp: "Formatted at 2026-09-25 10:12:03 on dist-test-slave-s6tm"
server_key: "ecf85e9ada4e3cf36f2c5ac4695fd440"
server_key_iv: "26a429a418a6b9f9ef1d5a7213b18fe2"
server_key_version: "encryptionkey@0"
I20260925 10:12:03.582819 1160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:12:03.583590 1175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:03.583767 1160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:03.583904 1160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "1ea9e59dbe9041c2844385a852e01909"
format_stamp: "Formatted at 2026-09-25 10:12:03 on dist-test-slave-s6tm"
server_key: "ecf85e9ada4e3cf36f2c5ac4695fd440"
server_key_iv: "26a429a418a6b9f9ef1d5a7213b18fe2"
server_key_version: "encryptionkey@0"
I20260925 10:12:03.584031 1160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:03.597509 1160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:03.599825 1160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:03.600040 1160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:03.604200 1160 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:39379
I20260925 10:12:03.604254 1227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:39379 every 8 connection(s)
I20260925 10:12:03.604636 1160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:03.605502 1228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:03.605868 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 1160
I20260925 10:12:03.605962 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:03.606177 14501 external_mini_cluster.cc:1428] Setting key c6d274b0f06416d9450670ee4375fe6a
I20260925 10:12:03.608575 1228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909: Bootstrap starting.
I20260925 10:12:03.609222 1228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:03.609587 1228 log.cc:824] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:03.610371 1228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909: No bootstrap required, opened a new log
Sep 25 10:12:03 dist-test-slave-s6tm krb5kdc[1147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:12:03.613072 1228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea9e59dbe9041c2844385a852e01909" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 39379 } }
I20260925 10:12:03.613233 1228 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:03.613281 1228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ea9e59dbe9041c2844385a852e01909, State: Initialized, Role: FOLLOWER
I20260925 10:12:03.613408 1228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea9e59dbe9041c2844385a852e01909" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 39379 } }
I20260925 10:12:03.613484 1228 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:03.613507 1228 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:03.613579 1228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:03.614161 1228 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea9e59dbe9041c2844385a852e01909" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 39379 } }
I20260925 10:12:03.614290 1228 leader_election.cc:302] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ea9e59dbe9041c2844385a852e01909; no voters:
I20260925 10:12:03.614471 1228 leader_election.cc:288] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:03.614507 1233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:03.614668 1233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909 [term 1 LEADER]: Becoming Leader. State: Replica: 1ea9e59dbe9041c2844385a852e01909, State: Running, Role: LEADER
I20260925 10:12:03.614773 1228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:03.614766 1233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea9e59dbe9041c2844385a852e01909" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 39379 } }
I20260925 10:12:03.615190 1234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ea9e59dbe9041c2844385a852e01909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea9e59dbe9041c2844385a852e01909" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 39379 } } }
I20260925 10:12:03.615345 1234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:03.615777 1242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:03.616031 1235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ea9e59dbe9041c2844385a852e01909. Latest consensus state: current_term: 1 leader_uuid: "1ea9e59dbe9041c2844385a852e01909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ea9e59dbe9041c2844385a852e01909" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 39379 } } }
I20260925 10:12:03.616278 1235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:03.616348 1242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:03.618007 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:03.606758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.14.41.126:39379)
0925 10:12:03.606939 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:03.606946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:03.606972 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:12:03.607376 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:03.607383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:03.607429 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:03.607546 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:03.608412 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:03.609294 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:03.610336 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:03.610808 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:03.610903 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:03.613808 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:03.613812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:03.613812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:03.613824 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:03.616338 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:03.616800 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:03.616803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:03.616804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:03.616906 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:03.617131 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:03.617132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:03.617132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:03.617385 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:03.617551 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:03.617664 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 10:12:03.617808 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260925 10:12:03.618242 1242 catalog_manager.cc:1347] Generated new cluster ID: b3dde0a0086a4ce7a3febc505d4c8f68
I20260925 10:12:03.618288 1242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:03.622651 1242 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:03.623181 1242 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:03.628047 1242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909: Generated new TSK 0
I20260925 10:12:03.628229 1242 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.14.41.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
Entry for principal kudu/127.14.41.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab.
I20260925 10:12:03.660912 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:39379
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:03.750854 1254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:03.751071 1254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:03.751096 1254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:03.752390 1254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:03.752439 1254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:03.752467 1254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:12:03.753995 1254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.65.keytab
--principal=kudu/127.14.41.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:39379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:03.754261 1254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:03.754521 1254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:03.759706 1262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:03.759727 1259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:03.759727 1260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:03.760100 1254 server_base.cc:1034] running on GCE node
I20260925 10:12:03.760294 1254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:03.760493 1254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:03.761674 1254 hybrid_clock.cc:648] HybridClock initialized: now 1790331123761656 us; error 26 us; skew 500 ppm
Sep 25 10:12:03 dist-test-slave-s6tm krb5kdc[1147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331123, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:03.764228 1254 init.cc:377] Logged in from keytab as kudu/127.14.41.65@KRBTEST.COM (short username kudu)
I20260925 10:12:03.764745 1254 webserver.cc:460] Webserver started at http://127.14.41.65:34513/ using document root <none> and password file <none>
I20260925 10:12:03.765002 1254 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:03.765076 1254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:03.765230 1254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:03.766338 1254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "b8e57be663e9479cb5cdc50c464f1af1"
format_stamp: "Formatted at 2026-09-25 10:12:03 on dist-test-slave-s6tm"
server_key: "bf72f91ced84e136b02a05d2ea6c60bf"
server_key_iv: "013bb7e6e8de95816f20bdb7fbc313cb"
server_key_version: "encryptionkey@0"
I20260925 10:12:03.766711 1254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "b8e57be663e9479cb5cdc50c464f1af1"
format_stamp: "Formatted at 2026-09-25 10:12:03 on dist-test-slave-s6tm"
server_key: "bf72f91ced84e136b02a05d2ea6c60bf"
server_key_iv: "013bb7e6e8de95816f20bdb7fbc313cb"
server_key_version: "encryptionkey@0"
I20260925 10:12:03.768187 1254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:12:03.769157 1269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:03.769497 1254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:12:03.769609 1254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "b8e57be663e9479cb5cdc50c464f1af1"
format_stamp: "Formatted at 2026-09-25 10:12:03 on dist-test-slave-s6tm"
server_key: "bf72f91ced84e136b02a05d2ea6c60bf"
server_key_iv: "013bb7e6e8de95816f20bdb7fbc313cb"
server_key_version: "encryptionkey@0"
I20260925 10:12:03.769690 1254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:03.791045 1254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:03.793732 1254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:03.793932 1254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:03.794194 1254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:03.794533 1254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:03.794589 1254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:03.794625 1254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:03.794673 1254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:03.801110 1254 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:44771
I20260925 10:12:03.801165 1382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:44771 every 8 connection(s)
I20260925 10:12:03.801505 1254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
Sep 25 10:12:03 dist-test-slave-s6tm krb5kdc[1147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331123, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.65@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:12:03.809528 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 1254
I20260925 10:12:03.809677 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:12:03.809890 14501 external_mini_cluster.cc:1428] Setting key 9558d336c7aecb1c9a002ff8c0464a95
I20260925 10:12:03.811770 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:03.802319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:43385 (local address 127.14.41.126:39379)
0925 10:12:03.802400 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:03.802422 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:03.802625 (+ 203us) server_negotiation.cc:407] Connection header received
0925 10:12:03.803172 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:03.803174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:03.803186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:03.803232 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:03.804286 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:03.804827 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:03.805857 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:03.806084 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:03.806175 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:03.808588 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:03.808591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:03.808591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:03.808602 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:03.810198 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:03.810712 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:03.810713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:03.810714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:03.810805 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:03.811054 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:03.811055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:03.811055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:03.811204 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:03.811278 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:03.811531 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 10:12:03.811628 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:12:03.811802 1385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:03.802291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:39379 (local address 127.14.41.65:43385)
0925 10:12:03.802489 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:03.802529 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:03.803058 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:03.803342 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:03.803346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:03.803949 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:03.804158 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:03.804163 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:03.804935 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:03.804937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:03.805757 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:03.805760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:03.806192 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:03.806193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:03.806331 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:03.806867 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:03.806876 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:03.808433 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:03.810286 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:03.810294 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:03.810296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:03.810618 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:03.810877 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:03.810878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:03.810879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:03.810985 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:03.811274 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:03.811280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:03.811400 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 10:12:03.811485 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 10:12:03.811613 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
I20260925 10:12:03.812481 1383 heartbeater.cc:347] Connected to a master server at 127.14.41.126:39379
I20260925 10:12:03.812587 1383 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:03.812788 1383 heartbeater.cc:511] Master 127.14.41.126:39379 requested a full tablet report, sending...
I20260925 10:12:03.813318 1192 ts_manager.cc:194] Registered new tserver with Master: b8e57be663e9479cb5cdc50c464f1af1 (127.14.41.65:44771)
I20260925 10:12:03.813927 1192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.65@KRBTEST.COM'} at 127.14.41.65:43385
WARNING: no policy specified for kudu/127.14.41.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
Entry for principal kudu/127.14.41.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab.
I20260925 10:12:03.838977 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:39379
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:03.923188 1388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:03.923425 1388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:03.923450 1388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:03.924701 1388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:03.924750 1388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:03.924777 1388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:12:03.926321 1388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.66.keytab
--principal=kudu/127.14.41.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:39379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:03.926587 1388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:03.926834 1388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:03.932258 1394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:03.932119 1396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:03.932425 1388 server_base.cc:1034] running on GCE node
W20260925 10:12:03.932120 1393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:03.932745 1388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:03.933056 1388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:03.934226 1388 hybrid_clock.cc:648] HybridClock initialized: now 1790331123934211 us; error 64 us; skew 500 ppm
Sep 25 10:12:03 dist-test-slave-s6tm krb5kdc[1147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331123, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:03.936970 1388 init.cc:377] Logged in from keytab as kudu/127.14.41.66@KRBTEST.COM (short username kudu)
I20260925 10:12:03.937525 1388 webserver.cc:460] Webserver started at http://127.14.41.66:34947/ using document root <none> and password file <none>
I20260925 10:12:03.937812 1388 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:03.937893 1388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:03.938068 1388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:03.939327 1388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "b12b7554e8ab48b2b9e30eaf60de8597"
format_stamp: "Formatted at 2026-09-25 10:12:03 on dist-test-slave-s6tm"
server_key: "0407c9811167872a29e0ca0e131f0c24"
server_key_iv: "1999aaa01fc115fe7b1d734e64f7cc92"
server_key_version: "encryptionkey@0"
I20260925 10:12:03.939739 1388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "b12b7554e8ab48b2b9e30eaf60de8597"
format_stamp: "Formatted at 2026-09-25 10:12:03 on dist-test-slave-s6tm"
server_key: "0407c9811167872a29e0ca0e131f0c24"
server_key_iv: "1999aaa01fc115fe7b1d734e64f7cc92"
server_key_version: "encryptionkey@0"
I20260925 10:12:03.941330 1388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:12:03.942230 1403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:03.942411 1388 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:12:03.942513 1388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "b12b7554e8ab48b2b9e30eaf60de8597"
format_stamp: "Formatted at 2026-09-25 10:12:03 on dist-test-slave-s6tm"
server_key: "0407c9811167872a29e0ca0e131f0c24"
server_key_iv: "1999aaa01fc115fe7b1d734e64f7cc92"
server_key_version: "encryptionkey@0"
I20260925 10:12:03.942648 1388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:03.977154 1388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:03.979673 1388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:03.979893 1388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:03.980212 1388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:03.980561 1388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:03.980618 1388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:03.980654 1388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:03.980686 1388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:03.986905 1388 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:38485
I20260925 10:12:03.986951 1517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:38485 every 8 connection(s)
I20260925 10:12:03.987301 1388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:12:03 dist-test-slave-s6tm krb5kdc[1147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331123, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.66@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:12:03.994539 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 1388
I20260925 10:12:03.994647 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:12:03.994850 14501 external_mini_cluster.cc:1428] Setting key 2e2de3ab3b4dad0003cae0243935260e
I20260925 10:12:03.997196 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:03.988152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:51619 (local address 127.14.41.126:39379)
0925 10:12:03.988196 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:03.988198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:03.988424 (+ 226us) server_negotiation.cc:407] Connection header received
0925 10:12:03.988911 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:03.988912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:03.988921 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:03.988981 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:03.989929 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:03.990420 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:03.991438 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:03.991630 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:03.991689 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:03.993886 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:03.993887 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:03.993888 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:03.993895 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:03.995630 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:03.996149 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:03.996150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:03.996150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:03.996211 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:03.996505 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:03.996506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:03.996506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:03.996633 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:03.996706 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:03.997013 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 10:12:03.997099 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:12:03.997439 1521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:03.988131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:39379 (local address 127.14.41.66:51619)
0925 10:12:03.988291 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:03.988335 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:03.988827 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:03.989045 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:03.989048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:03.989642 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:03.989848 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:03.989853 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:03.990509 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:03.990510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:03.991353 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:03.991356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:03.991716 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:03.991716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:03.991831 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:03.992345 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:03.992354 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:03.993791 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:03.995712 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:03.995724 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:03.995726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:03.996070 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:03.996305 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:03.996306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:03.996306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:03.996434 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:03.996710 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:03.996718 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:03.996873 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 10:12:03.996981 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:12:03.997166 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260925 10:12:03.997974 1519 heartbeater.cc:347] Connected to a master server at 127.14.41.126:39379
I20260925 10:12:03.998098 1519 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:03.998344 1519 heartbeater.cc:511] Master 127.14.41.126:39379 requested a full tablet report, sending...
I20260925 10:12:03.998890 1192 ts_manager.cc:194] Registered new tserver with Master: b12b7554e8ab48b2b9e30eaf60de8597 (127.14.41.66:38485)
I20260925 10:12:03.999332 1192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.66@KRBTEST.COM'} at 127.14.41.66:51619
WARNING: no policy specified for kudu/127.14.41.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
Entry for principal kudu/127.14.41.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab.
I20260925 10:12:04.024294 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:39379
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:04.108685 1524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:04.108897 1524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:04.108923 1524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:04.110181 1524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:04.110249 1524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:04.110280 1524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:12:04.111771 1524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.67.keytab
--principal=kudu/127.14.41.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:39379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:04.112028 1524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:04.112296 1524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:04.117834 1530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:04.117854 1532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:04.117969 1524 server_base.cc:1034] running on GCE node
W20260925 10:12:04.117827 1529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:04.118289 1524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:04.118552 1524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:04.119722 1524 hybrid_clock.cc:648] HybridClock initialized: now 1790331124119692 us; error 44 us; skew 500 ppm
Sep 25 10:12:04 dist-test-slave-s6tm krb5kdc[1147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331124, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:04.122499 1524 init.cc:377] Logged in from keytab as kudu/127.14.41.67@KRBTEST.COM (short username kudu)
I20260925 10:12:04.123124 1524 webserver.cc:460] Webserver started at http://127.14.41.67:38227/ using document root <none> and password file <none>
I20260925 10:12:04.123375 1524 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:04.123447 1524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:04.123615 1524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:04.124822 1524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "bb29565751d84b37a7d870cc6a13735f"
format_stamp: "Formatted at 2026-09-25 10:12:04 on dist-test-slave-s6tm"
server_key: "d1fce4bd0b49bbcf49540aa1cbf5db2b"
server_key_iv: "ff72e7dfcbf6793cd2a1622a960105a2"
server_key_version: "encryptionkey@0"
I20260925 10:12:04.125214 1524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "bb29565751d84b37a7d870cc6a13735f"
format_stamp: "Formatted at 2026-09-25 10:12:04 on dist-test-slave-s6tm"
server_key: "d1fce4bd0b49bbcf49540aa1cbf5db2b"
server_key_iv: "ff72e7dfcbf6793cd2a1622a960105a2"
server_key_version: "encryptionkey@0"
I20260925 10:12:04.126624 1524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:12:04.127377 1539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:04.127566 1524 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 10:12:04.127692 1524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "bb29565751d84b37a7d870cc6a13735f"
format_stamp: "Formatted at 2026-09-25 10:12:04 on dist-test-slave-s6tm"
server_key: "d1fce4bd0b49bbcf49540aa1cbf5db2b"
server_key_iv: "ff72e7dfcbf6793cd2a1622a960105a2"
server_key_version: "encryptionkey@0"
I20260925 10:12:04.127770 1524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:04.145625 1524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:04.148510 1524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:04.148737 1524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:04.149034 1524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:04.149403 1524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:04.149466 1524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:04.149513 1524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:04.149561 1524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:04.156190 1524 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:42985
I20260925 10:12:04.156265 1653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:42985 every 8 connection(s)
I20260925 10:12:04.156613 1524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:12:04.159530 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 1524
I20260925 10:12:04.159643 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:12:04.159842 14501 external_mini_cluster.cc:1428] Setting key fbd6ce97216391e5637e208be1dff101
Sep 25 10:12:04 dist-test-slave-s6tm krb5kdc[1147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331124, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.67@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:12:04.166970 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.157399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:49521 (local address 127.14.41.126:39379)
0925 10:12:04.157450 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:04.157453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:04.157648 (+ 195us) server_negotiation.cc:407] Connection header received
0925 10:12:04.158226 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:04.158227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:04.158238 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:04.158324 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:04.159367 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.160136 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.161482 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.161667 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.161727 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.163994 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:04.163995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:04.163996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:04.164004 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:04.165522 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:04.165936 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:04.165937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:04.165937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:04.165990 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:04.166283 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:04.166284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:04.166284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:04.166424 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:04.166502 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:04.166826 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 10:12:04.166902 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:12:04.167349 1656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.157336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:39379 (local address 127.14.41.67:49521)
0925 10:12:04.157526 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:04.157572 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:04.158115 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:04.158393 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:04.158397 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:04.159043 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:04.159255 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.159261 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.160235 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.160236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.161400 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.161403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.161767 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.161768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.161885 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.162362 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:04.162370 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:04.163847 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:04.165610 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:04.165617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:04.165619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:04.165866 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:04.166054 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:04.166055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:04.166055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:04.166165 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:04.166537 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:04.166543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:04.166671 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 10:12:04.166779 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:12:04.166914 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260925 10:12:04.167996 1654 heartbeater.cc:347] Connected to a master server at 127.14.41.126:39379
I20260925 10:12:04.168217 1654 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:04.168437 1654 heartbeater.cc:511] Master 127.14.41.126:39379 requested a full tablet report, sending...
I20260925 10:12:04.168898 1192 ts_manager.cc:194] Registered new tserver with Master: bb29565751d84b37a7d870cc6a13735f (127.14.41.67:42985)
I20260925 10:12:04.169281 1192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.14.41.67@KRBTEST.COM'} at 127.14.41.67:49521
I20260925 10:12:04.171566 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:12:04.179689 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.173030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.14.41.126:39379)
0925 10:12:04.173125 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:04.173127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:04.173332 (+ 205us) server_negotiation.cc:407] Connection header received
0925 10:12:04.173384 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:04.173384 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:04.173393 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:04.173444 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:04.173968 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.174442 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.175203 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.175389 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.175440 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.176764 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:04.176765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:04.176766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:04.176773 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:04.178358 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:04.178714 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:04.178715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:04.178715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:04.178768 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:04.178958 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:04.178958 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:04.178959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:04.179306 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:04.179390 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:04.179551 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 10:12:04.179621 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:12:04.181420 1192 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:12:04.183157 1192 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:12:04.190394 1666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.185751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:38485 (local address 127.0.0.1:34074)
0925 10:12:04.186111 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:04.186159 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:04.186234 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:04.186797 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:04.186798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:04.186801 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:04.186848 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.186850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.188262 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.188263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.189235 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.189238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.190053 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.190054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.190139 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.190155 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 10:12:04.190240 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 10:12:04.190295 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":9,"thread_start_us":81,"threads_started":1}
I20260925 10:12:04.190392 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.185874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34074 (local address 127.14.41.66:38485)
0925 10:12:04.186168 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:04.186174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:04.186216 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:12:04.186364 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:04.186367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:04.186383 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:04.186697 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:04.186939 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.188060 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.189329 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.189963 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.190040 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.190097 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:04.190264 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 10:12:04.190308 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":103,"threads_started":1}
I20260925 10:12:04.190963 1670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.185751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:44771 (local address 127.0.0.1:33488)
0925 10:12:04.186118 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:04.186159 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:04.186233 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:04.186578 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:04.186582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:04.186584 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:04.186660 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.186661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.188051 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.188052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.189626 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.189628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.190696 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.190697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.190768 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.190773 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:12:04.190826 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 10:12:04.190873 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":69,"thread_start_us":81,"threads_started":1}
I20260925 10:12:04.191138 1667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.185973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:42985 (local address 127.0.0.1:42456)
0925 10:12:04.186369 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:04.186378 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:04.186422 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:04.186665 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:04.186666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:04.186668 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:04.186711 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.186712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.188214 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.188215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.189625 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.189628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.190810 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.190811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.190875 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.190879 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:12:04.190935 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 10:12:04.190977 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":107,"thread_start_us":78,"threads_started":1}
I20260925 10:12:04.191274 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.186102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42456 (local address 127.14.41.67:42985)
0925 10:12:04.186354 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:04.186360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:04.186415 (+ 55us) server_negotiation.cc:407] Connection header received
0925 10:12:04.186486 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:04.186488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:04.186503 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:04.186577 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:04.186772 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.188024 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.189748 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.190736 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.190813 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.190878 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:04.190949 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 10:12:04.191002 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":84,"threads_started":1}
I20260925 10:12:04.191274 1669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.185964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33488 (local address 127.14.41.65:44771)
0925 10:12:04.186405 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:04.186409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:04.186419 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:12:04.186455 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:04.186457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:04.186471 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:04.186547 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:04.186813 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.187905 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.189725 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.190618 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.190686 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.190749 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:04.191142 (+ 393us) server_negotiation.cc:299] Negotiation successful
0925 10:12:04.191190 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":80,"threads_started":1}
I20260925 10:12:04.191226 1452 tablet_service.cc:1467] Processing CreateTablet for tablet 89f86193e20749dba41eeba7925e9fd5 (DEFAULT_TABLE table=test-table [id=e37acca3959048f69ec8455b02b7f146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:04.191635 1452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f86193e20749dba41eeba7925e9fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:04.191910 1317 tablet_service.cc:1467] Processing CreateTablet for tablet 89f86193e20749dba41eeba7925e9fd5 (DEFAULT_TABLE table=test-table [id=e37acca3959048f69ec8455b02b7f146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:04.192322 1317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f86193e20749dba41eeba7925e9fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:04.194571 1672 tablet_bootstrap.cc:492] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597: Bootstrap starting.
I20260925 10:12:04.194974 1673 tablet_bootstrap.cc:492] T 89f86193e20749dba41eeba7925e9fd5 P b8e57be663e9479cb5cdc50c464f1af1: Bootstrap starting.
I20260925 10:12:04.195144 1587 tablet_service.cc:1467] Processing CreateTablet for tablet 89f86193e20749dba41eeba7925e9fd5 (DEFAULT_TABLE table=test-table [id=e37acca3959048f69ec8455b02b7f146]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:04.195454 1672 tablet_bootstrap.cc:654] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:04.195489 1587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f86193e20749dba41eeba7925e9fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:04.195721 1673 tablet_bootstrap.cc:654] T 89f86193e20749dba41eeba7925e9fd5 P b8e57be663e9479cb5cdc50c464f1af1: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:04.196053 1672 log.cc:824] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:04.196442 1673 log.cc:824] T 89f86193e20749dba41eeba7925e9fd5 P b8e57be663e9479cb5cdc50c464f1af1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:04.196934 1672 tablet_bootstrap.cc:492] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597: No bootstrap required, opened a new log
I20260925 10:12:04.197019 1672 ts_tablet_manager.cc:1397] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:12:04.197237 1673 tablet_bootstrap.cc:492] T 89f86193e20749dba41eeba7925e9fd5 P b8e57be663e9479cb5cdc50c464f1af1: No bootstrap required, opened a new log
I20260925 10:12:04.197324 1673 ts_tablet_manager.cc:1397] T 89f86193e20749dba41eeba7925e9fd5 P b8e57be663e9479cb5cdc50c464f1af1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:12:04.198910 1676 tablet_bootstrap.cc:492] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f: Bootstrap starting.
I20260925 10:12:04.198880 1673 raft_consensus.cc:348] T 89f86193e20749dba41eeba7925e9fd5 P b8e57be663e9479cb5cdc50c464f1af1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e57be663e9479cb5cdc50c464f1af1" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44771 } } peers { permanent_uuid: "bb29565751d84b37a7d870cc6a13735f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42985 } } peers { permanent_uuid: "b12b7554e8ab48b2b9e30eaf60de8597" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38485 } }
I20260925 10:12:04.199095 1673 raft_consensus.cc:374] T 89f86193e20749dba41eeba7925e9fd5 P b8e57be663e9479cb5cdc50c464f1af1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:04.199143 1673 raft_consensus.cc:729] T 89f86193e20749dba41eeba7925e9fd5 P b8e57be663e9479cb5cdc50c464f1af1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e57be663e9479cb5cdc50c464f1af1, State: Initialized, Role: FOLLOWER
I20260925 10:12:04.199265 1673 consensus_queue.cc:260] T 89f86193e20749dba41eeba7925e9fd5 P b8e57be663e9479cb5cdc50c464f1af1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e57be663e9479cb5cdc50c464f1af1" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44771 } } peers { permanent_uuid: "bb29565751d84b37a7d870cc6a13735f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42985 } } peers { permanent_uuid: "b12b7554e8ab48b2b9e30eaf60de8597" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38485 } }
I20260925 10:12:04.199180 1672 raft_consensus.cc:348] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e57be663e9479cb5cdc50c464f1af1" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44771 } } peers { permanent_uuid: "bb29565751d84b37a7d870cc6a13735f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42985 } } peers { permanent_uuid: "b12b7554e8ab48b2b9e30eaf60de8597" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38485 } }
I20260925 10:12:04.199357 1672 raft_consensus.cc:374] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:04.199404 1672 raft_consensus.cc:729] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b12b7554e8ab48b2b9e30eaf60de8597, State: Initialized, Role: FOLLOWER
I20260925 10:12:04.199545 1672 consensus_queue.cc:260] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e57be663e9479cb5cdc50c464f1af1" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44771 } } peers { permanent_uuid: "bb29565751d84b37a7d870cc6a13735f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42985 } } peers { permanent_uuid: "b12b7554e8ab48b2b9e30eaf60de8597" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38485 } }
I20260925 10:12:04.199626 1383 heartbeater.cc:503] Master 127.14.41.126:39379 was elected leader, sending a full tablet report...
I20260925 10:12:04.199609 1673 ts_tablet_manager.cc:1428] T 89f86193e20749dba41eeba7925e9fd5 P b8e57be663e9479cb5cdc50c464f1af1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:12:04.199820 1672 ts_tablet_manager.cc:1428] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:12:04.199847 1676 tablet_bootstrap.cc:654] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:04.199985 1519 heartbeater.cc:503] Master 127.14.41.126:39379 was elected leader, sending a full tablet report...
I20260925 10:12:04.200526 1676 log.cc:824] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:04.201452 1676 tablet_bootstrap.cc:492] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f: No bootstrap required, opened a new log
I20260925 10:12:04.201508 1676 ts_tablet_manager.cc:1397] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:12:04.203135 1676 raft_consensus.cc:348] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e57be663e9479cb5cdc50c464f1af1" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44771 } } peers { permanent_uuid: "bb29565751d84b37a7d870cc6a13735f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42985 } } peers { permanent_uuid: "b12b7554e8ab48b2b9e30eaf60de8597" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38485 } }
I20260925 10:12:04.203270 1676 raft_consensus.cc:374] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:04.203292 1676 raft_consensus.cc:729] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb29565751d84b37a7d870cc6a13735f, State: Initialized, Role: FOLLOWER
I20260925 10:12:04.203398 1676 consensus_queue.cc:260] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e57be663e9479cb5cdc50c464f1af1" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44771 } } peers { permanent_uuid: "bb29565751d84b37a7d870cc6a13735f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42985 } } peers { permanent_uuid: "b12b7554e8ab48b2b9e30eaf60de8597" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38485 } }
I20260925 10:12:04.203621 1676 ts_tablet_manager.cc:1428] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:12:04.203747 1654 heartbeater.cc:503] Master 127.14.41.126:39379 was elected leader, sending a full tablet report...
W20260925 10:12:04.237891 1520 tablet.cc:2369] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:12:04.302385 1384 tablet.cc:2369] T 89f86193e20749dba41eeba7925e9fd5 P b8e57be663e9479cb5cdc50c464f1af1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:12:04.316733 1678 raft_consensus.cc:482] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:12:04.316869 1678 raft_consensus.cc:504] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e57be663e9479cb5cdc50c464f1af1" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44771 } } peers { permanent_uuid: "bb29565751d84b37a7d870cc6a13735f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42985 } } peers { permanent_uuid: "b12b7554e8ab48b2b9e30eaf60de8597" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38485 } }
I20260925 10:12:04.317425 1678 leader_election.cc:288] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8e57be663e9479cb5cdc50c464f1af1 (127.14.41.65:44771), bb29565751d84b37a7d870cc6a13735f (127.14.41.67:42985)
I20260925 10:12:04.320533 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.317901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:48655 (local address 127.14.41.67:42985)
0925 10:12:04.317943 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:04.317945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:04.318111 (+ 166us) server_negotiation.cc:407] Connection header received
0925 10:12:04.318142 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:04.318143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:04.318154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:04.318207 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:04.318364 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.318861 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.319763 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.320180 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.320266 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.320274 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:04.320435 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 10:12:04.320474 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:12:04.320529 1669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.317697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:56541 (local address 127.14.41.65:44771)
0925 10:12:04.317776 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:04.317779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:04.317849 (+ 70us) server_negotiation.cc:407] Connection header received
0925 10:12:04.317937 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:04.317938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:04.317951 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:04.318003 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:04.318222 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.318771 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.319730 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.320167 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.320226 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.320251 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:04.320404 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 10:12:04.320451 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:12:04.320529 1521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.317686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:44771 (local address 127.14.41.66:56541)
0925 10:12:04.317749 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:04.317763 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:04.317866 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:04.318017 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:04.318018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:04.318020 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:04.318115 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.318118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.318852 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.318853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.319630 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.319632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.320239 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.320240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.320334 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.320337 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:12:04.320383 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 10:12:04.320448 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":1}
I20260925 10:12:04.320547 1681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.317833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:42985 (local address 127.14.41.66:48655)
0925 10:12:04.318005 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:04.318015 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:04.318067 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:04.318219 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:04.318220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:04.318222 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:04.318294 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.318296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.318942 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.318942 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.319653 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.319654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.320276 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.320277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.320346 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.320349 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:12:04.320419 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 10:12:04.320463 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":4,"thread_start_us":117,"threads_started":1}
I20260925 10:12:04.320983 1608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f86193e20749dba41eeba7925e9fd5" candidate_uuid: "b12b7554e8ab48b2b9e30eaf60de8597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb29565751d84b37a7d870cc6a13735f" is_pre_election: true
I20260925 10:12:04.320982 1337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f86193e20749dba41eeba7925e9fd5" candidate_uuid: "b12b7554e8ab48b2b9e30eaf60de8597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8e57be663e9479cb5cdc50c464f1af1" is_pre_election: true
I20260925 10:12:04.321153 1337 raft_consensus.cc:2454] T 89f86193e20749dba41eeba7925e9fd5 P b8e57be663e9479cb5cdc50c464f1af1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b12b7554e8ab48b2b9e30eaf60de8597 in term 0.
I20260925 10:12:04.321148 1608 raft_consensus.cc:2454] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b12b7554e8ab48b2b9e30eaf60de8597 in term 0.
I20260925 10:12:04.321393 1405 leader_election.cc:302] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b12b7554e8ab48b2b9e30eaf60de8597, b8e57be663e9479cb5cdc50c464f1af1; no voters:
I20260925 10:12:04.321552 1678 raft_consensus.cc:2781] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:12:04.321619 1678 raft_consensus.cc:482] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:12:04.321688 1678 raft_consensus.cc:3037] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:04.322459 1678 raft_consensus.cc:504] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e57be663e9479cb5cdc50c464f1af1" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44771 } } peers { permanent_uuid: "bb29565751d84b37a7d870cc6a13735f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42985 } } peers { permanent_uuid: "b12b7554e8ab48b2b9e30eaf60de8597" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38485 } }
I20260925 10:12:04.322626 1678 leader_election.cc:288] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [CANDIDATE]: Term 1 election: Requested vote from peers b8e57be663e9479cb5cdc50c464f1af1 (127.14.41.65:44771), bb29565751d84b37a7d870cc6a13735f (127.14.41.67:42985)
I20260925 10:12:04.322814 1337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f86193e20749dba41eeba7925e9fd5" candidate_uuid: "b12b7554e8ab48b2b9e30eaf60de8597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8e57be663e9479cb5cdc50c464f1af1"
I20260925 10:12:04.322832 1608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f86193e20749dba41eeba7925e9fd5" candidate_uuid: "b12b7554e8ab48b2b9e30eaf60de8597" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb29565751d84b37a7d870cc6a13735f"
I20260925 10:12:04.322904 1337 raft_consensus.cc:3037] T 89f86193e20749dba41eeba7925e9fd5 P b8e57be663e9479cb5cdc50c464f1af1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:04.322913 1608 raft_consensus.cc:3037] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:04.323642 1337 raft_consensus.cc:2454] T 89f86193e20749dba41eeba7925e9fd5 P b8e57be663e9479cb5cdc50c464f1af1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b12b7554e8ab48b2b9e30eaf60de8597 in term 1.
I20260925 10:12:04.323742 1608 raft_consensus.cc:2454] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b12b7554e8ab48b2b9e30eaf60de8597 in term 1.
I20260925 10:12:04.323892 1405 leader_election.cc:302] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b12b7554e8ab48b2b9e30eaf60de8597, b8e57be663e9479cb5cdc50c464f1af1; no voters:
I20260925 10:12:04.324007 1678 raft_consensus.cc:2781] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:04.324184 1678 raft_consensus.cc:686] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [term 1 LEADER]: Becoming Leader. State: Replica: b12b7554e8ab48b2b9e30eaf60de8597, State: Running, Role: LEADER
I20260925 10:12:04.324287 1678 consensus_queue.cc:237] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e57be663e9479cb5cdc50c464f1af1" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44771 } } peers { permanent_uuid: "bb29565751d84b37a7d870cc6a13735f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42985 } } peers { permanent_uuid: "b12b7554e8ab48b2b9e30eaf60de8597" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38485 } }
I20260925 10:12:04.325138 1191 catalog_manager.cc:5495] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 reported cstate change: term changed from 0 to 1, leader changed from <none> to b12b7554e8ab48b2b9e30eaf60de8597 (127.14.41.66). New cstate: current_term: 1 leader_uuid: "b12b7554e8ab48b2b9e30eaf60de8597" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e57be663e9479cb5cdc50c464f1af1" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb29565751d84b37a7d870cc6a13735f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b12b7554e8ab48b2b9e30eaf60de8597" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 38485 } health_report { overall_health: HEALTHY } } }
I20260925 10:12:04.344354 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.341777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34084 (local address 127.14.41.66:38485)
0925 10:12:04.341841 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:04.341843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:04.341851 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:12:04.341904 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:04.341905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:04.341928 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:04.342002 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:04.342148 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.342652 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.343453 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.343653 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.343749 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.343961 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:04.344078 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:04.344177 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:04.344197 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 10:12:04.344220 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 10:12:04.346247 1608 raft_consensus.cc:1261] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f [term 1 FOLLOWER]: Refusing update from remote peer b12b7554e8ab48b2b9e30eaf60de8597: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:04.346254 1337 raft_consensus.cc:1261] T 89f86193e20749dba41eeba7925e9fd5 P b8e57be663e9479cb5cdc50c464f1af1 [term 1 FOLLOWER]: Refusing update from remote peer b12b7554e8ab48b2b9e30eaf60de8597: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:04.346678 1682 consensus_queue.cc:1035] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb29565751d84b37a7d870cc6a13735f" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 42985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:04.346835 1678 consensus_queue.cc:1035] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8e57be663e9479cb5cdc50c464f1af1" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:04.350207 1686 mvcc.cc:205] Tried to move back new op lower bound from 7333196285319483392 to 7333196285232951296. Current Snapshot: MvccSnapshot[applied={T|T < 7333196285319483392 or (T in {7333196285319483392})}]
I20260925 10:12:04.350371 1687 mvcc.cc:205] Tried to move back new op lower bound from 7333196285319483392 to 7333196285232951296. Current Snapshot: MvccSnapshot[applied={T|T < 7333196285319483392}]
I20260925 10:12:04.351754 1192 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:12:04.351851 1192 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:12:04.352653 1192 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ea9e59dbe9041c2844385a852e01909: Sending DeleteTablet for 3 replicas of tablet 89f86193e20749dba41eeba7925e9fd5
I20260925 10:12:04.353006 1452 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f86193e20749dba41eeba7925e9fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:04 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:34074
I20260925 10:12:04.353061 1317 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f86193e20749dba41eeba7925e9fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:04 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:33488
I20260925 10:12:04.353245 1695 tablet_replica.cc:330] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597: stopping tablet replica
I20260925 10:12:04.353286 1587 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f86193e20749dba41eeba7925e9fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:04 UTC) from {username='kudu', principal='kudu/127.14.41.126@KRBTEST.COM'} at 127.0.0.1:42456
I20260925 10:12:04.353343 1695 raft_consensus.cc:2229] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:12:04.353451 1695 raft_consensus.cc:2258] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:04.353492 1696 tablet_replica.cc:330] T 89f86193e20749dba41eeba7925e9fd5 P b8e57be663e9479cb5cdc50c464f1af1: stopping tablet replica
I20260925 10:12:04.353502 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 1254
I20260925 10:12:04.353581 1696 raft_consensus.cc:2229] T 89f86193e20749dba41eeba7925e9fd5 P b8e57be663e9479cb5cdc50c464f1af1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:12:04.353672 1696 raft_consensus.cc:2258] T 89f86193e20749dba41eeba7925e9fd5 P b8e57be663e9479cb5cdc50c464f1af1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:04.353802 1697 tablet_replica.cc:330] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f: stopping tablet replica
I20260925 10:12:04.353881 1697 raft_consensus.cc:2229] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:12:04.354034 1697 raft_consensus.cc:2258] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:04.354480 1695 ts_tablet_manager.cc:1905] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:04.354903 1697 ts_tablet_manager.cc:1905] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:04.356318 1695 ts_tablet_manager.cc:1918] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:12:04.356405 1695 log.cc:1196] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/89f86193e20749dba41eeba7925e9fd5
I20260925 10:12:04.356618 1697 ts_tablet_manager.cc:1918] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:12:04.356732 1697 log.cc:1196] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/89f86193e20749dba41eeba7925e9fd5
I20260925 10:12:04.356752 1695 ts_tablet_manager.cc:1939] T 89f86193e20749dba41eeba7925e9fd5 P b12b7554e8ab48b2b9e30eaf60de8597: Deleting consensus metadata
I20260925 10:12:04.357039 1697 ts_tablet_manager.cc:1939] T 89f86193e20749dba41eeba7925e9fd5 P bb29565751d84b37a7d870cc6a13735f: Deleting consensus metadata
I20260925 10:12:04.357218 1179 catalog_manager.cc:4841] TS b12b7554e8ab48b2b9e30eaf60de8597 (127.14.41.66:38485): tablet 89f86193e20749dba41eeba7925e9fd5 (table test-table [id=e37acca3959048f69ec8455b02b7f146]) successfully deleted
I20260925 10:12:04.357465 1177 catalog_manager.cc:4841] TS bb29565751d84b37a7d870cc6a13735f (127.14.41.67:42985): tablet 89f86193e20749dba41eeba7925e9fd5 (table test-table [id=e37acca3959048f69ec8455b02b7f146]) successfully deleted
I20260925 10:12:04.360600 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 1388
W20260925 10:12:04.363186 1177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.41.65:44771 (error 108)
I20260925 10:12:04.363955 1667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.363751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:44771 (local address 127.0.0.1:33496)
0925 10:12:04.363840 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:04.363890 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.65:44771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260925 10:12:04.364081 1177 catalog_manager.cc:4584] TS b8e57be663e9479cb5cdc50c464f1af1 (127.14.41.65:44771): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 89f86193e20749dba41eeba7925e9fd5: Network error: Client connection negotiation failed: client connection to 127.14.41.65:44771: connect: Connection refused (error 111)
I20260925 10:12:04.368010 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 1524
I20260925 10:12:04.376461 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 1160
2026-09-25T10:12:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2104 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T10:12:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:04Z Disabled control of system clock
I20260925 10:12:04.410738 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:33537
--webserver_interface=127.14.41.126
--webserver_port=0
--builtin_ntp_servers=127.14.41.84:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:33537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:12:04.492894 1702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:04.493119 1702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:04.493139 1702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:04.494254 1702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:04.494293 1702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:04.494306 1702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:04.494318 1702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:04.494331 1702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:04.495759 1702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:33537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:33537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:04.496043 1702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:04.496304 1702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:04.501267 1710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:04.501259 1708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:04.501286 1702 server_base.cc:1034] running on GCE node
W20260925 10:12:04.501259 1707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:04.501789 1702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:04.502034 1702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:04.503168 1702 hybrid_clock.cc:648] HybridClock initialized: now 1790331124503145 us; error 39 us; skew 500 ppm
I20260925 10:12:04.504431 1702 webserver.cc:460] Webserver started at http://127.14.41.126:37155/ using document root <none> and password file <none>
I20260925 10:12:04.504637 1702 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:04.504709 1702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:04.504813 1702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:04.505870 1702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "5db42f87ccdd44388adfe18aadfe9b7d"
format_stamp: "Formatted at 2026-09-25 10:12:04 on dist-test-slave-s6tm"
server_key: "33fb1a2d1386cceb215ab2b43caa4d19"
server_key_iv: "033a452a8b115488d9ec09a70024dabf"
server_key_version: "encryptionkey@0"
I20260925 10:12:04.506174 1702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "5db42f87ccdd44388adfe18aadfe9b7d"
format_stamp: "Formatted at 2026-09-25 10:12:04 on dist-test-slave-s6tm"
server_key: "33fb1a2d1386cceb215ab2b43caa4d19"
server_key_iv: "033a452a8b115488d9ec09a70024dabf"
server_key_version: "encryptionkey@0"
I20260925 10:12:04.507464 1702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:12:04.508213 1716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:04.508379 1702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:12:04.508453 1702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "5db42f87ccdd44388adfe18aadfe9b7d"
format_stamp: "Formatted at 2026-09-25 10:12:04 on dist-test-slave-s6tm"
server_key: "33fb1a2d1386cceb215ab2b43caa4d19"
server_key_iv: "033a452a8b115488d9ec09a70024dabf"
server_key_version: "encryptionkey@0"
I20260925 10:12:04.508502 1702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:04.523540 1702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:04.523842 1702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:04.524021 1702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:04.528131 1702 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:33537
I20260925 10:12:04.528184 1769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:33537 every 8 connection(s)
I20260925 10:12:04.528553 1702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:04.529577 1770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:04.532119 1770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d: Bootstrap starting.
I20260925 10:12:04.532753 1770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:04.533069 1770 log.cc:824] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:04.533766 1770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d: No bootstrap required, opened a new log
I20260925 10:12:04.535393 1770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db42f87ccdd44388adfe18aadfe9b7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 33537 } }
I20260925 10:12:04.535568 1770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:04.535609 1770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5db42f87ccdd44388adfe18aadfe9b7d, State: Initialized, Role: FOLLOWER
I20260925 10:12:04.535671 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 1702
I20260925 10:12:04.535789 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:04.535812 1770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db42f87ccdd44388adfe18aadfe9b7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 33537 } }
I20260925 10:12:04.535945 1770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:04.535992 1770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:04.536043 1770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:04.536062 14501 external_mini_cluster.cc:1428] Setting key 19d1300739ace6c10b70989e16806733
I20260925 10:12:04.536851 1770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db42f87ccdd44388adfe18aadfe9b7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 33537 } }
I20260925 10:12:04.536965 1770 leader_election.cc:302] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5db42f87ccdd44388adfe18aadfe9b7d; no voters:
I20260925 10:12:04.537117 1770 leader_election.cc:288] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:04.537182 1775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:04.537358 1775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d [term 1 LEADER]: Becoming Leader. State: Replica: 5db42f87ccdd44388adfe18aadfe9b7d, State: Running, Role: LEADER
I20260925 10:12:04.537478 1775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db42f87ccdd44388adfe18aadfe9b7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 33537 } }
I20260925 10:12:04.537370 1770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:04.537830 1775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5db42f87ccdd44388adfe18aadfe9b7d. Latest consensus state: current_term: 1 leader_uuid: "5db42f87ccdd44388adfe18aadfe9b7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db42f87ccdd44388adfe18aadfe9b7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 33537 } } }
I20260925 10:12:04.537990 1775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:04.538496 1782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:04.539063 1782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:04.539022 1776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5db42f87ccdd44388adfe18aadfe9b7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5db42f87ccdd44388adfe18aadfe9b7d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 33537 } } }
I20260925 10:12:04.539539 1776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:04.540719 1782 catalog_manager.cc:1347] Generated new cluster ID: 4cecec6a5fb14d628bcbb03c02b91d11
I20260925 10:12:04.540773 1782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:04.542732 1774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.536626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55100 (local address 127.14.41.126:33537)
0925 10:12:04.536889 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:04.536899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:04.536938 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:12:04.537063 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:04.537072 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:04.537150 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:04.537311 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:04.537669 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.539388 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.540513 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.541125 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.541244 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.542016 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:04.542020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:04.542020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:04.542221 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:04.542260 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:04.542274 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:04.542340 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:04.542440 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 10:12:04.542518 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":110,"threads_started":1}
I20260925 10:12:04.555009 1782 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:04.555600 1782 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:04.566783 1782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d: Generated new TSK 0
I20260925 10:12:04.567018 1782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:04.569880 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.65:0
--local_ip_for_outbound_sockets=127.14.41.65
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:33537
--builtin_ntp_servers=127.14.41.84:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:12:04.660914 1794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:04.661126 1794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:04.661155 1794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:04.662319 1794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:04.662359 1794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:04.662384 1794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.65
I20260925 10:12:04.663757 1794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.41.65
--webserver_port=0
--tserver_master_addrs=127.14.41.126:33537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.65
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:04.664021 1794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:04.664331 1794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:04.669384 1803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:04.669625 1794 server_base.cc:1034] running on GCE node
W20260925 10:12:04.669389 1801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:04.669409 1800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:04.669970 1794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:04.670217 1794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:04.671348 1794 hybrid_clock.cc:648] HybridClock initialized: now 1790331124671335 us; error 28 us; skew 500 ppm
I20260925 10:12:04.672885 1794 webserver.cc:460] Webserver started at http://127.14.41.65:37927/ using document root <none> and password file <none>
I20260925 10:12:04.673159 1794 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:04.673209 1794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:04.673341 1794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:04.674484 1794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-0/data/instance:
uuid: "8dedc671ca4e4936a1d812a8441c91b7"
format_stamp: "Formatted at 2026-09-25 10:12:04 on dist-test-slave-s6tm"
server_key: "1e1414bdc0b9f4869bd4eeeef630fc46"
server_key_iv: "37c3c219d7ba98985ba735a86e03d3aa"
server_key_version: "encryptionkey@0"
I20260925 10:12:04.674875 1794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance:
uuid: "8dedc671ca4e4936a1d812a8441c91b7"
format_stamp: "Formatted at 2026-09-25 10:12:04 on dist-test-slave-s6tm"
server_key: "1e1414bdc0b9f4869bd4eeeef630fc46"
server_key_iv: "37c3c219d7ba98985ba735a86e03d3aa"
server_key_version: "encryptionkey@0"
I20260925 10:12:04.676385 1794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:12:04.677196 1809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:04.677407 1794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:12:04.677490 1794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-0/wal
uuid: "8dedc671ca4e4936a1d812a8441c91b7"
format_stamp: "Formatted at 2026-09-25 10:12:04 on dist-test-slave-s6tm"
server_key: "1e1414bdc0b9f4869bd4eeeef630fc46"
server_key_iv: "37c3c219d7ba98985ba735a86e03d3aa"
server_key_version: "encryptionkey@0"
I20260925 10:12:04.677565 1794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:04.687060 1794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:04.687335 1794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:04.687520 1794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:04.687747 1794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:04.688131 1794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:04.688185 1794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:04.688218 1794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:04.688249 1794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:04.694518 1794 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.65:34799
I20260925 10:12:04.694588 1923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.65:34799 every 8 connection(s)
I20260925 10:12:04.694981 1794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-0/data/info.pb
I20260925 10:12:04.700645 1774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.695765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.65:58573 (local address 127.14.41.126:33537)
0925 10:12:04.695824 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:04.695827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:04.696081 (+ 254us) server_negotiation.cc:407] Connection header received
0925 10:12:04.696672 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:04.696674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:04.696691 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:04.696755 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:04.697506 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.698100 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.699117 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.699336 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.699429 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.700365 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:04.700367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:04.700367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:04.700373 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:04.700390 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:04.700394 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:04.700443 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:04.700518 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 10:12:04.700557 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 10:12:04.700870 1926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.695734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:33537 (local address 127.14.41.65:58573)
0925 10:12:04.695913 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:04.695989 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:04.696516 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:04.696828 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:04.696832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:04.696843 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:04.697408 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.697414 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.698191 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.698192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.699022 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.699025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.699440 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.699441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.699585 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.700205 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:12:04.700205 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:04.700235 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:12:04.700259 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:04.700449 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:04.700454 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:04.700457 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:12:04.700523 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 10:12:04.700636 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260925 10:12:04.701464 1924 heartbeater.cc:347] Connected to a master server at 127.14.41.126:33537
I20260925 10:12:04.701562 1924 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:04.701750 1924 heartbeater.cc:511] Master 127.14.41.126:33537 requested a full tablet report, sending...
I20260925 10:12:04.702401 1734 ts_manager.cc:194] Registered new tserver with Master: 8dedc671ca4e4936a1d812a8441c91b7 (127.14.41.65:34799)
I20260925 10:12:04.703171 1734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.41.65:58573
I20260925 10:12:04.705190 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 1794
I20260925 10:12:04.705277 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-0/wal/instance
I20260925 10:12:04.705436 14501 external_mini_cluster.cc:1428] Setting key 343e3e97ea93deacb1fec4c4dc1ad66c
I20260925 10:12:04.706331 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.66:0
--local_ip_for_outbound_sockets=127.14.41.66
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:33537
--builtin_ntp_servers=127.14.41.84:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:12:04.790827 1927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:04.791054 1927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:04.791076 1927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:04.792421 1927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:04.792469 1927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:04.792495 1927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.66
I20260925 10:12:04.793887 1927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.41.66
--webserver_port=0
--tserver_master_addrs=127.14.41.126:33537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.66
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:04.794160 1927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:04.794468 1927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:04.799705 1932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:04.799705 1935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:04.799713 1927 server_base.cc:1034] running on GCE node
W20260925 10:12:04.799713 1933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:04.800215 1927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:04.800473 1927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:04.801657 1927 hybrid_clock.cc:648] HybridClock initialized: now 1790331124801648 us; error 37 us; skew 500 ppm
I20260925 10:12:04.802919 1927 webserver.cc:460] Webserver started at http://127.14.41.66:40703/ using document root <none> and password file <none>
I20260925 10:12:04.803246 1927 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:04.803349 1927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:04.803493 1927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:04.804690 1927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-1/data/instance:
uuid: "adb79ae36608480a84867552fc428ea3"
format_stamp: "Formatted at 2026-09-25 10:12:04 on dist-test-slave-s6tm"
server_key: "94c91742b794b99dfd1ce624e181338e"
server_key_iv: "f27851ad68016b9f013cc5ce48ce152c"
server_key_version: "encryptionkey@0"
I20260925 10:12:04.805068 1927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance:
uuid: "adb79ae36608480a84867552fc428ea3"
format_stamp: "Formatted at 2026-09-25 10:12:04 on dist-test-slave-s6tm"
server_key: "94c91742b794b99dfd1ce624e181338e"
server_key_iv: "f27851ad68016b9f013cc5ce48ce152c"
server_key_version: "encryptionkey@0"
I20260925 10:12:04.806423 1927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:12:04.807180 1942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:04.807374 1927 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:12:04.807472 1927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-1/wal
uuid: "adb79ae36608480a84867552fc428ea3"
format_stamp: "Formatted at 2026-09-25 10:12:04 on dist-test-slave-s6tm"
server_key: "94c91742b794b99dfd1ce624e181338e"
server_key_iv: "f27851ad68016b9f013cc5ce48ce152c"
server_key_version: "encryptionkey@0"
I20260925 10:12:04.807556 1927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:04.819898 1927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:04.820305 1927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:04.820459 1927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:04.820712 1927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:04.821065 1927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:04.821118 1927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:04.821159 1927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:04.821185 1927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:04.827337 1927 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.66:35893
I20260925 10:12:04.827386 2056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.66:35893 every 8 connection(s)
I20260925 10:12:04.827766 1927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-1/data/info.pb
I20260925 10:12:04.831563 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 1927
I20260925 10:12:04.831712 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-1/wal/instance
I20260925 10:12:04.831961 14501 external_mini_cluster.cc:1428] Setting key bee33d689dbe93b7d736cc0ecbab19a4
I20260925 10:12:04.832893 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.41.67:0
--local_ip_for_outbound_sockets=127.14.41.67
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:33537
--builtin_ntp_servers=127.14.41.84:45295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:12:04.834827 1774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.828745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.66:51265 (local address 127.14.41.126:33537)
0925 10:12:04.828799 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:04.828802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:04.829175 (+ 373us) server_negotiation.cc:407] Connection header received
0925 10:12:04.830234 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:04.830236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:04.830249 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:04.830311 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:04.831297 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.832157 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.833312 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.833524 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.833607 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.834536 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:04.834538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:04.834538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:04.834543 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:04.834560 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:04.834565 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:04.834614 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:04.834698 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 10:12:04.834740 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:12:04.835052 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.828736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:33537 (local address 127.14.41.66:51265)
0925 10:12:04.829037 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:04.829083 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:04.829821 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:04.830452 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:04.830456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:04.830467 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:04.831172 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.831180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.832260 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.832262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.833203 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.833206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.833620 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.833621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.833771 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.834374 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:12:04.834374 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:04.834406 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:12:04.834430 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:04.834620 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:04.834626 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:04.834629 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:12:04.834693 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 10:12:04.834805 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":84,"threads_started":1}
I20260925 10:12:04.835512 2057 heartbeater.cc:347] Connected to a master server at 127.14.41.126:33537
I20260925 10:12:04.835626 2057 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:04.835832 2057 heartbeater.cc:511] Master 127.14.41.126:33537 requested a full tablet report, sending...
I20260925 10:12:04.836258 1734 ts_manager.cc:194] Registered new tserver with Master: adb79ae36608480a84867552fc428ea3 (127.14.41.66:35893)
I20260925 10:12:04.836630 1734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.41.66:51265
W20260925 10:12:04.916445 2060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:04.916673 2060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:04.916698 2060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:04.917959 2060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:04.918002 2060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:04.918030 2060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.41.67
I20260925 10:12:04.919451 2060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:45295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.41.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.41.67
--webserver_port=0
--tserver_master_addrs=127.14.41.126:33537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.41.67
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:04.919709 2060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:04.920032 2060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:04.925009 2065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:04.925019 2068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:04.925024 2066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:04.925268 2060 server_base.cc:1034] running on GCE node
I20260925 10:12:04.925511 2060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:04.925709 2060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:04.926851 2060 hybrid_clock.cc:648] HybridClock initialized: now 1790331124926838 us; error 31 us; skew 500 ppm
I20260925 10:12:04.928290 2060 webserver.cc:460] Webserver started at http://127.14.41.67:46625/ using document root <none> and password file <none>
I20260925 10:12:04.928551 2060 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:04.928599 2060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:04.928730 2060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:04.929833 2060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-2/data/instance:
uuid: "3d8a30e004e04a369dc82a53ba3f84bc"
format_stamp: "Formatted at 2026-09-25 10:12:04 on dist-test-slave-s6tm"
server_key: "8a9347d1e3080dfbf9c14c67f0f1b1b7"
server_key_iv: "f5e42d8872aec67046ded5161da2ca4e"
server_key_version: "encryptionkey@0"
I20260925 10:12:04.930194 2060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance:
uuid: "3d8a30e004e04a369dc82a53ba3f84bc"
format_stamp: "Formatted at 2026-09-25 10:12:04 on dist-test-slave-s6tm"
server_key: "8a9347d1e3080dfbf9c14c67f0f1b1b7"
server_key_iv: "f5e42d8872aec67046ded5161da2ca4e"
server_key_version: "encryptionkey@0"
I20260925 10:12:04.931528 2060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:12:04.932503 2074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:04.932714 2060 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:12:04.932812 2060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-2/wal
uuid: "3d8a30e004e04a369dc82a53ba3f84bc"
format_stamp: "Formatted at 2026-09-25 10:12:04 on dist-test-slave-s6tm"
server_key: "8a9347d1e3080dfbf9c14c67f0f1b1b7"
server_key_iv: "f5e42d8872aec67046ded5161da2ca4e"
server_key_version: "encryptionkey@0"
I20260925 10:12:04.932904 2060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:04.945854 2060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:04.946190 2060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:04.946332 2060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:04.946557 2060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:04.946908 2060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:04.946962 2060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:04.947005 2060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:04.947032 2060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:04.952922 2060 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.67:39373
I20260925 10:12:04.953030 2189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.67:39373 every 8 connection(s)
I20260925 10:12:04.953354 2060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-2/data/info.pb
I20260925 10:12:04.958334 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 2060
I20260925 10:12:04.958467 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-2/wal/instance
I20260925 10:12:04.958717 14501 external_mini_cluster.cc:1428] Setting key a0b96dfbc92227d1d3eb664ddadb9b9d
I20260925 10:12:04.959209 1774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.954131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:49859 (local address 127.14.41.126:33537)
0925 10:12:04.954203 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:04.954205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:04.954471 (+ 266us) server_negotiation.cc:407] Connection header received
0925 10:12:04.954966 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:04.954967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:04.954978 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:04.955036 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:04.955836 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.956451 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.957430 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.957618 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.957697 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.958788 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:04.958789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:04.958789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:04.958796 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:04.958822 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:04.958833 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:04.958902 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:04.958984 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 10:12:04.959016 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:12:04.959354 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.954137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:33537 (local address 127.14.41.67:49859)
0925 10:12:04.954348 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:04.954385 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:04.954864 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:04.955196 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:04.955199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:04.955208 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:04.955706 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.955710 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.956539 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.956540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.957336 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.957339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.957723 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.957724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.957844 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.958599 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:12:04.958600 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:04.958646 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:12:04.958676 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:04.958907 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:04.958913 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:04.958916 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:12:04.958984 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 10:12:04.959111 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260925 10:12:04.959805 2190 heartbeater.cc:347] Connected to a master server at 127.14.41.126:33537
I20260925 10:12:04.959897 2190 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:04.960106 2190 heartbeater.cc:511] Master 127.14.41.126:33537 requested a full tablet report, sending...
I20260925 10:12:04.960414 1734 ts_manager.cc:194] Registered new tserver with Master: 3d8a30e004e04a369dc82a53ba3f84bc (127.14.41.67:39373)
I20260925 10:12:04.960773 1734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.41.67:49859
I20260925 10:12:04.971095 14501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:12:04.975806 1774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.972739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55106 (local address 127.14.41.126:33537)
0925 10:12:04.972825 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:04.972828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:04.972932 (+ 104us) server_negotiation.cc:407] Connection header received
0925 10:12:04.973012 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:04.973013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:04.973027 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:04.973080 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:04.973246 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.973746 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.974575 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.974784 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.974865 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.975527 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:04.975528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:04.975529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:04.975535 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:04.975552 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:04.975556 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:04.975627 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:04.975679 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 10:12:04.975711 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 10:12:04.977702 1734 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:12:04.979506 1734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:12:04.986330 2203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.982363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45346 (local address 127.14.41.67:39373)
0925 10:12:04.982557 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:04.982561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:04.982573 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:12:04.982615 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:04.982617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:04.982632 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:04.982737 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:04.982884 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.983741 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.985154 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.985911 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.985994 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.986095 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:04.986200 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 10:12:04.986239 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":81,"threads_started":1}
I20260925 10:12:04.986346 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.981993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:34799 (local address 127.0.0.1:49880)
0925 10:12:04.982127 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:04.982183 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:04.982257 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:04.982634 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:04.982637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:04.982640 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:04.982719 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.982721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.984054 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.984055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.985072 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.985074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.986016 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.986017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.986099 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.986114 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 10:12:04.986164 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 10:12:04.986230 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":2,"thread_start_us":129,"threads_started":1}
I20260925 10:12:04.986349 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.981993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.67:39373 (local address 127.0.0.1:45346)
0925 10:12:04.982401 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:04.982412 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:04.982459 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:04.982748 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:04.982749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:04.982751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:04.982800 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.982802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.983827 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.983828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.985049 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.985052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.986012 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.986013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.986098 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.986114 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 10:12:04.986186 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 10:12:04.986240 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":6,"thread_start_us":221,"threads_started":1}
I20260925 10:12:04.986680 2200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.981998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:35893 (local address 127.0.0.1:39916)
0925 10:12:04.982469 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:04.982481 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:04.982540 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:04.983091 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:04.983092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:04.983094 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:04.983129 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.983130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.984470 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.984471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.985402 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:04.985404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.986250 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.986252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:04.986311 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.986314 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:12:04.986361 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 10:12:04.986403 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":77,"threads_started":1}
I20260925 10:12:04.987003 2201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.982191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.14.41.66:35893)
0925 10:12:04.982867 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:04.982872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:04.982881 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:12:04.982922 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:04.982924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:04.982939 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:04.983008 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:04.983217 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.984347 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.985491 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.986032 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.986110 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.986217 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:04.986447 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 10:12:04.986925 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":106,"threads_started":1}
I20260925 10:12:04.987013 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:04.982270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.14.41.65:34799)
0925 10:12:04.982453 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:04.982458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:04.982470 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:12:04.982515 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:04.982517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:04.982533 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:04.982631 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:04.982807 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.983955 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.985183 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:04.985916 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:04.986664 (+ 748us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:04.986841 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:04.986866 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 10:12:04.986913 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":98,"threads_started":1}
I20260925 10:12:04.987313 2123 tablet_service.cc:1467] Processing CreateTablet for tablet 26cc2b3915af4df5b016f34f8d4973dd (DEFAULT_TABLE table=test-table [id=e358aa32ad9a4c2ba254cea9aed4fa40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:04.987706 2123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26cc2b3915af4df5b016f34f8d4973dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:04.987622 1990 tablet_service.cc:1467] Processing CreateTablet for tablet 26cc2b3915af4df5b016f34f8d4973dd (DEFAULT_TABLE table=test-table [id=e358aa32ad9a4c2ba254cea9aed4fa40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:04.987963 1990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26cc2b3915af4df5b016f34f8d4973dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:04.988003 1857 tablet_service.cc:1467] Processing CreateTablet for tablet 26cc2b3915af4df5b016f34f8d4973dd (DEFAULT_TABLE table=test-table [id=e358aa32ad9a4c2ba254cea9aed4fa40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:04.988346 1857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26cc2b3915af4df5b016f34f8d4973dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:04.990989 2204 tablet_bootstrap.cc:492] T 26cc2b3915af4df5b016f34f8d4973dd P 8dedc671ca4e4936a1d812a8441c91b7: Bootstrap starting.
I20260925 10:12:04.991377 2206 tablet_bootstrap.cc:492] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3: Bootstrap starting.
I20260925 10:12:04.991678 2204 tablet_bootstrap.cc:654] T 26cc2b3915af4df5b016f34f8d4973dd P 8dedc671ca4e4936a1d812a8441c91b7: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:04.992013 2206 tablet_bootstrap.cc:654] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:04.992082 2204 log.cc:824] T 26cc2b3915af4df5b016f34f8d4973dd P 8dedc671ca4e4936a1d812a8441c91b7: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:04.992100 2205 tablet_bootstrap.cc:492] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc: Bootstrap starting.
I20260925 10:12:04.992365 2206 log.cc:824] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:04.992698 2205 tablet_bootstrap.cc:654] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:04.992890 2204 tablet_bootstrap.cc:492] T 26cc2b3915af4df5b016f34f8d4973dd P 8dedc671ca4e4936a1d812a8441c91b7: No bootstrap required, opened a new log
I20260925 10:12:04.992967 2204 ts_tablet_manager.cc:1397] T 26cc2b3915af4df5b016f34f8d4973dd P 8dedc671ca4e4936a1d812a8441c91b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:12:04.992988 2205 log.cc:824] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:04.993726 2205 tablet_bootstrap.cc:492] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc: No bootstrap required, opened a new log
I20260925 10:12:04.993781 2205 ts_tablet_manager.cc:1397] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:12:04.995036 2204 raft_consensus.cc:348] T 26cc2b3915af4df5b016f34f8d4973dd P 8dedc671ca4e4936a1d812a8441c91b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8a30e004e04a369dc82a53ba3f84bc" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39373 } } peers { permanent_uuid: "adb79ae36608480a84867552fc428ea3" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35893 } } peers { permanent_uuid: "8dedc671ca4e4936a1d812a8441c91b7" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34799 } }
I20260925 10:12:04.995203 2204 raft_consensus.cc:374] T 26cc2b3915af4df5b016f34f8d4973dd P 8dedc671ca4e4936a1d812a8441c91b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:04.995250 2204 raft_consensus.cc:729] T 26cc2b3915af4df5b016f34f8d4973dd P 8dedc671ca4e4936a1d812a8441c91b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dedc671ca4e4936a1d812a8441c91b7, State: Initialized, Role: FOLLOWER
I20260925 10:12:04.995378 2204 consensus_queue.cc:260] T 26cc2b3915af4df5b016f34f8d4973dd P 8dedc671ca4e4936a1d812a8441c91b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8a30e004e04a369dc82a53ba3f84bc" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39373 } } peers { permanent_uuid: "adb79ae36608480a84867552fc428ea3" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35893 } } peers { permanent_uuid: "8dedc671ca4e4936a1d812a8441c91b7" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34799 } }
I20260925 10:12:04.995692 2204 ts_tablet_manager.cc:1428] T 26cc2b3915af4df5b016f34f8d4973dd P 8dedc671ca4e4936a1d812a8441c91b7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:12:04.995740 1924 heartbeater.cc:503] Master 127.14.41.126:33537 was elected leader, sending a full tablet report...
I20260925 10:12:04.995863 2205 raft_consensus.cc:348] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8a30e004e04a369dc82a53ba3f84bc" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39373 } } peers { permanent_uuid: "adb79ae36608480a84867552fc428ea3" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35893 } } peers { permanent_uuid: "8dedc671ca4e4936a1d812a8441c91b7" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34799 } }
I20260925 10:12:04.996063 2205 raft_consensus.cc:374] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:04.996109 2205 raft_consensus.cc:729] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8a30e004e04a369dc82a53ba3f84bc, State: Initialized, Role: FOLLOWER
I20260925 10:12:04.996248 2205 consensus_queue.cc:260] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8a30e004e04a369dc82a53ba3f84bc" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39373 } } peers { permanent_uuid: "adb79ae36608480a84867552fc428ea3" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35893 } } peers { permanent_uuid: "8dedc671ca4e4936a1d812a8441c91b7" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34799 } }
I20260925 10:12:04.996510 2205 ts_tablet_manager.cc:1428] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:12:04.996559 2206 tablet_bootstrap.cc:492] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3: No bootstrap required, opened a new log
I20260925 10:12:04.996556 2190 heartbeater.cc:503] Master 127.14.41.126:33537 was elected leader, sending a full tablet report...
I20260925 10:12:04.996623 2206 ts_tablet_manager.cc:1397] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 10:12:04.998795 2206 raft_consensus.cc:348] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8a30e004e04a369dc82a53ba3f84bc" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39373 } } peers { permanent_uuid: "adb79ae36608480a84867552fc428ea3" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35893 } } peers { permanent_uuid: "8dedc671ca4e4936a1d812a8441c91b7" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34799 } }
I20260925 10:12:04.998925 2206 raft_consensus.cc:374] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:04.998947 2206 raft_consensus.cc:729] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb79ae36608480a84867552fc428ea3, State: Initialized, Role: FOLLOWER
I20260925 10:12:04.999058 2206 consensus_queue.cc:260] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8a30e004e04a369dc82a53ba3f84bc" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39373 } } peers { permanent_uuid: "adb79ae36608480a84867552fc428ea3" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35893 } } peers { permanent_uuid: "8dedc671ca4e4936a1d812a8441c91b7" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34799 } }
I20260925 10:12:04.999298 2206 ts_tablet_manager.cc:1428] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 10:12:04.999375 2057 heartbeater.cc:503] Master 127.14.41.126:33537 was elected leader, sending a full tablet report...
W20260925 10:12:05.078385 2058 tablet.cc:2369] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:12:05.106933 2211 raft_consensus.cc:482] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:12:05.107066 2211 raft_consensus.cc:504] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8a30e004e04a369dc82a53ba3f84bc" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39373 } } peers { permanent_uuid: "adb79ae36608480a84867552fc428ea3" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35893 } } peers { permanent_uuid: "8dedc671ca4e4936a1d812a8441c91b7" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34799 } }
I20260925 10:12:05.107486 2211 leader_election.cc:288] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adb79ae36608480a84867552fc428ea3 (127.14.41.66:35893), 8dedc671ca4e4936a1d812a8441c91b7 (127.14.41.65:34799)
I20260925 10:12:05.111644 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:05.107723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:34799 (local address 127.14.41.67:46655)
0925 10:12:05.107777 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:05.107788 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:05.107834 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:05.108043 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:05.108044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:05.108046 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:05.108128 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:05.108130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:05.109225 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:05.109226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:05.110533 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:05.110536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:05.111309 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:05.111311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:05.111391 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:05.111397 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:12:05.111454 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 10:12:05.111503 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":28}
I20260925 10:12:05.111671 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:05.107812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:46655 (local address 127.14.41.65:34799)
0925 10:12:05.107881 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:05.107883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:05.107893 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:12:05.107947 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:05.107948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:05.107958 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:05.108013 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:05.108285 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:05.109140 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:05.110628 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:05.111222 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:05.111302 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:05.111310 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:05.111569 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 10:12:05.111607 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:12:05.111671 2201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:05.107823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.14.41.67:42599 (local address 127.14.41.66:35893)
0925 10:12:05.107869 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:05.107871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:05.108171 (+ 300us) server_negotiation.cc:407] Connection header received
0925 10:12:05.108347 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:05.108350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:05.108368 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:05.108437 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:05.108577 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:05.109264 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:05.110655 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:05.111224 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:05.111302 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:05.111310 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:05.111500 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 10:12:05.111565 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:12:05.111657 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:05.107733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.66:35893 (local address 127.14.41.67:42599)
0925 10:12:05.108074 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:05.108085 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:05.108161 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:05.108455 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:05.108456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:05.108457 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:05.108507 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:05.108508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:05.109327 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:05.109328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:05.110533 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:05.110536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:05.111319 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:05.111319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:05.111391 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:05.111397 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:12:05.111485 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 10:12:05.111513 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":156,"thread_start_us":122,"threads_started":1}
I20260925 10:12:05.112124 1878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26cc2b3915af4df5b016f34f8d4973dd" candidate_uuid: "3d8a30e004e04a369dc82a53ba3f84bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dedc671ca4e4936a1d812a8441c91b7" is_pre_election: true
I20260925 10:12:05.112310 1878 raft_consensus.cc:2454] T 26cc2b3915af4df5b016f34f8d4973dd P 8dedc671ca4e4936a1d812a8441c91b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8a30e004e04a369dc82a53ba3f84bc in term 0.
I20260925 10:12:05.112440 2011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26cc2b3915af4df5b016f34f8d4973dd" candidate_uuid: "3d8a30e004e04a369dc82a53ba3f84bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adb79ae36608480a84867552fc428ea3" is_pre_election: true
I20260925 10:12:05.112550 2075 leader_election.cc:302] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8a30e004e04a369dc82a53ba3f84bc, 8dedc671ca4e4936a1d812a8441c91b7; no voters:
I20260925 10:12:05.112597 2011 raft_consensus.cc:2454] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8a30e004e04a369dc82a53ba3f84bc in term 0.
I20260925 10:12:05.112720 2211 raft_consensus.cc:2781] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:12:05.112802 2211 raft_consensus.cc:482] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:12:05.112833 2211 raft_consensus.cc:3037] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:05.113538 2211 raft_consensus.cc:504] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8a30e004e04a369dc82a53ba3f84bc" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39373 } } peers { permanent_uuid: "adb79ae36608480a84867552fc428ea3" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35893 } } peers { permanent_uuid: "8dedc671ca4e4936a1d812a8441c91b7" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34799 } }
I20260925 10:12:05.113687 2211 leader_election.cc:288] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [CANDIDATE]: Term 1 election: Requested vote from peers adb79ae36608480a84867552fc428ea3 (127.14.41.66:35893), 8dedc671ca4e4936a1d812a8441c91b7 (127.14.41.65:34799)
I20260925 10:12:05.113828 2011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26cc2b3915af4df5b016f34f8d4973dd" candidate_uuid: "3d8a30e004e04a369dc82a53ba3f84bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adb79ae36608480a84867552fc428ea3"
I20260925 10:12:05.113850 1878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26cc2b3915af4df5b016f34f8d4973dd" candidate_uuid: "3d8a30e004e04a369dc82a53ba3f84bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dedc671ca4e4936a1d812a8441c91b7"
I20260925 10:12:05.113904 2011 raft_consensus.cc:3037] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:05.113957 1878 raft_consensus.cc:3037] T 26cc2b3915af4df5b016f34f8d4973dd P 8dedc671ca4e4936a1d812a8441c91b7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:05.114706 1878 raft_consensus.cc:2454] T 26cc2b3915af4df5b016f34f8d4973dd P 8dedc671ca4e4936a1d812a8441c91b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8a30e004e04a369dc82a53ba3f84bc in term 1.
I20260925 10:12:05.114708 2011 raft_consensus.cc:2454] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8a30e004e04a369dc82a53ba3f84bc in term 1.
I20260925 10:12:05.114881 2077 leader_election.cc:302] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8a30e004e04a369dc82a53ba3f84bc, adb79ae36608480a84867552fc428ea3; no voters:
I20260925 10:12:05.114981 2211 raft_consensus.cc:2781] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:05.115129 2211 raft_consensus.cc:686] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [term 1 LEADER]: Becoming Leader. State: Replica: 3d8a30e004e04a369dc82a53ba3f84bc, State: Running, Role: LEADER
I20260925 10:12:05.115226 2211 consensus_queue.cc:237] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8a30e004e04a369dc82a53ba3f84bc" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39373 } } peers { permanent_uuid: "adb79ae36608480a84867552fc428ea3" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35893 } } peers { permanent_uuid: "8dedc671ca4e4936a1d812a8441c91b7" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34799 } }
I20260925 10:12:05.116123 1733 catalog_manager.cc:5495] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d8a30e004e04a369dc82a53ba3f84bc (127.14.41.67). New cstate: current_term: 1 leader_uuid: "3d8a30e004e04a369dc82a53ba3f84bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8a30e004e04a369dc82a53ba3f84bc" member_type: VOTER last_known_addr { host: "127.14.41.67" port: 39373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adb79ae36608480a84867552fc428ea3" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dedc671ca4e4936a1d812a8441c91b7" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34799 } health_report { overall_health: UNKNOWN } } }
I20260925 10:12:05.130800 2203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:05.128329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45362 (local address 127.14.41.67:39373)
0925 10:12:05.128378 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:05.128380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:05.128390 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:12:05.128476 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:05.128477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:05.128495 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:05.128573 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:05.128714 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:05.129216 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:05.130001 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:05.130193 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:05.130267 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:05.130450 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:05.130548 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:05.130642 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:05.130661 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 10:12:05.130678 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:12:05.132699 1878 raft_consensus.cc:1261] T 26cc2b3915af4df5b016f34f8d4973dd P 8dedc671ca4e4936a1d812a8441c91b7 [term 1 FOLLOWER]: Refusing update from remote peer 3d8a30e004e04a369dc82a53ba3f84bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:05.132697 2011 raft_consensus.cc:1261] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3 [term 1 FOLLOWER]: Refusing update from remote peer 3d8a30e004e04a369dc82a53ba3f84bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:05.133114 2214 consensus_queue.cc:1035] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "adb79ae36608480a84867552fc428ea3" member_type: VOTER last_known_addr { host: "127.14.41.66" port: 35893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:05.133230 2211 consensus_queue.cc:1035] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dedc671ca4e4936a1d812a8441c91b7" member_type: VOTER last_known_addr { host: "127.14.41.65" port: 34799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:05.136117 2219 mvcc.cc:205] Tried to move back new op lower bound from 7333196288540831744 to 7333196288472625152. Current Snapshot: MvccSnapshot[applied={T|T < 7333196288540831744}]
I20260925 10:12:05.136222 2218 mvcc.cc:205] Tried to move back new op lower bound from 7333196288540831744 to 7333196288472625152. Current Snapshot: MvccSnapshot[applied={T|T < 7333196288540831744}]
I20260925 10:12:05.137880 1734 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:12:05.138003 1734 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:12:05.138741 1734 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5db42f87ccdd44388adfe18aadfe9b7d: Sending DeleteTablet for 3 replicas of tablet 26cc2b3915af4df5b016f34f8d4973dd
I20260925 10:12:05.139109 1857 tablet_service.cc:1514] Processing DeleteTablet for tablet 26cc2b3915af4df5b016f34f8d4973dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:05 UTC) from {username='slave'} at 127.0.0.1:49880
I20260925 10:12:05.139235 1990 tablet_service.cc:1514] Processing DeleteTablet for tablet 26cc2b3915af4df5b016f34f8d4973dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:05 UTC) from {username='slave'} at 127.0.0.1:39916
I20260925 10:12:05.139434 2123 tablet_service.cc:1514] Processing DeleteTablet for tablet 26cc2b3915af4df5b016f34f8d4973dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:05 UTC) from {username='slave'} at 127.0.0.1:45346
I20260925 10:12:05.139456 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 1794
I20260925 10:12:05.140766 2229 tablet_replica.cc:330] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc: stopping tablet replica
I20260925 10:12:05.140874 2229 raft_consensus.cc:2229] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:12:05.140971 2229 raft_consensus.cc:2258] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:05.141117 2228 tablet_replica.cc:330] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3: stopping tablet replica
I20260925 10:12:05.141215 2228 raft_consensus.cc:2229] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:12:05.141319 2228 raft_consensus.cc:2258] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:05.141713 2229 ts_tablet_manager.cc:1905] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:05.141871 2228 ts_tablet_manager.cc:1905] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:05.143930 2228 ts_tablet_manager.cc:1918] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:12:05.143963 2229 ts_tablet_manager.cc:1918] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:12:05.144032 2229 log.cc:1196] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-2/wal/wals/26cc2b3915af4df5b016f34f8d4973dd
I20260925 10:12:05.144044 2228 log.cc:1196] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3: Deleting WAL directory at /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790330877480195-14501-0/minicluster-data/ts-1/wal/wals/26cc2b3915af4df5b016f34f8d4973dd
I20260925 10:12:05.144301 2228 ts_tablet_manager.cc:1939] T 26cc2b3915af4df5b016f34f8d4973dd P adb79ae36608480a84867552fc428ea3: Deleting consensus metadata
I20260925 10:12:05.144414 2229 ts_tablet_manager.cc:1939] T 26cc2b3915af4df5b016f34f8d4973dd P 3d8a30e004e04a369dc82a53ba3f84bc: Deleting consensus metadata
I20260925 10:12:05.144826 1719 catalog_manager.cc:4841] TS adb79ae36608480a84867552fc428ea3 (127.14.41.66:35893): tablet 26cc2b3915af4df5b016f34f8d4973dd (table test-table [id=e358aa32ad9a4c2ba254cea9aed4fa40]) successfully deleted
I20260925 10:12:05.144834 1718 catalog_manager.cc:4841] TS 3d8a30e004e04a369dc82a53ba3f84bc (127.14.41.67:39373): tablet 26cc2b3915af4df5b016f34f8d4973dd (table test-table [id=e358aa32ad9a4c2ba254cea9aed4fa40]) successfully deleted
W20260925 10:12:05.151065 1717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.14.41.65:34799 (error 108)
I20260925 10:12:05.151630 2200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:05.151480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.65:34799 (local address 127.0.0.1:49882)
0925 10:12:05.151543 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:05.151592 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.65:34799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 10:12:05.151697 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 1927
W20260925 10:12:05.153318 1717 catalog_manager.cc:4584] TS 8dedc671ca4e4936a1d812a8441c91b7 (127.14.41.65:34799): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 26cc2b3915af4df5b016f34f8d4973dd: Network error: Client connection negotiation failed: client connection to 127.14.41.65:34799: connect: Connection refused (error 111)
I20260925 10:12:05.156654 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 2060
I20260925 10:12:05.161422 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 1702
2026-09-25T10:12:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (783 ms)
[----------] 6 tests from EncryptionPolicyTest (8335 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:05 dist-test-slave-s6tm krb5kdc[2233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:05 dist-test-slave-s6tm krb5kdc[2233](info): set up 2 sockets
Sep 25 10:12:05 dist-test-slave-s6tm krb5kdc[2233](info): commencing operation
krb5kdc: starting...
W20260925 10:12:06.277499 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:06 dist-test-slave-s6tm krb5kdc[2233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:12:06.404223 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39183
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:06.487614 2246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:06.487896 2246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:06.487972 2246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:06.489234 2246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:06.489319 2246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:06.489341 2246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:06.489362 2246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:06.489392 2246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:06.490983 2246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39183
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:06.491297 2246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:06.491534 2246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:06.496783 2252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:06.496783 2254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:06.496814 2246 server_base.cc:1034] running on GCE node
W20260925 10:12:06.496783 2251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:06.497464 2246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:06.497762 2246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:06.498921 2246 hybrid_clock.cc:648] HybridClock initialized: now 1790331126498908 us; error 31 us; skew 500 ppm
Sep 25 10:12:06 dist-test-slave-s6tm krb5kdc[2233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:06.505786 2246 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:12:06.506282 2246 webserver.cc:460] Webserver started at http://127.0.0.1:40331/ using document root <none> and password file <none>
I20260925 10:12:06.506523 2246 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:06.506573 2246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:06.506699 2246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:06.508208 2246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "e806fa1438b3421aaff4ce83def08148"
format_stamp: "Formatted at 2026-09-25 10:12:06 on dist-test-slave-s6tm"
server_key: "a396ec7e84b3dd008057ed6aab5cdef8"
server_key_iv: "a24cd87ff1c491a3754fbb4fafdb55f5"
server_key_version: "encryptionkey@0"
I20260925 10:12:06.508622 2246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "e806fa1438b3421aaff4ce83def08148"
format_stamp: "Formatted at 2026-09-25 10:12:06 on dist-test-slave-s6tm"
server_key: "a396ec7e84b3dd008057ed6aab5cdef8"
server_key_iv: "a24cd87ff1c491a3754fbb4fafdb55f5"
server_key_version: "encryptionkey@0"
I20260925 10:12:06.510088 2246 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 10:12:06.511031 2261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:06.511238 2246 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:06.511348 2246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "e806fa1438b3421aaff4ce83def08148"
format_stamp: "Formatted at 2026-09-25 10:12:06 on dist-test-slave-s6tm"
server_key: "a396ec7e84b3dd008057ed6aab5cdef8"
server_key_iv: "a24cd87ff1c491a3754fbb4fafdb55f5"
server_key_version: "encryptionkey@0"
I20260925 10:12:06.511430 2246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:06.523527 2246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:06.526270 2246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:06.526474 2246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:06.531066 2246 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39183
I20260925 10:12:06.531100 2313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39183 every 8 connection(s)
I20260925 10:12:06.531530 2246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:06.532660 2314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:06.534996 2314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148: Bootstrap starting.
I20260925 10:12:06.535579 2314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:06.535821 2314 log.cc:824] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:06.536597 2314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148: No bootstrap required, opened a new log
I20260925 10:12:06.538381 2314 raft_consensus.cc:348] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e806fa1438b3421aaff4ce83def08148" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39183 } }
I20260925 10:12:06.538537 2314 raft_consensus.cc:374] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:06.538564 2314 raft_consensus.cc:729] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e806fa1438b3421aaff4ce83def08148, State: Initialized, Role: FOLLOWER
I20260925 10:12:06.538727 2314 consensus_queue.cc:260] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e806fa1438b3421aaff4ce83def08148" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39183 } }
I20260925 10:12:06.538831 2314 raft_consensus.cc:388] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:06.538856 2314 raft_consensus.cc:482] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:06.538895 2314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:06.539458 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 2246
I20260925 10:12:06.539464 2314 raft_consensus.cc:504] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e806fa1438b3421aaff4ce83def08148" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39183 } }
I20260925 10:12:06.539593 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:06.539589 2314 leader_election.cc:302] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e806fa1438b3421aaff4ce83def08148; no voters:
I20260925 10:12:06.539781 2314 leader_election.cc:288] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:06.539810 2317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:06.539860 14501 external_mini_cluster.cc:1428] Setting key 89bcc654ae99f72aaa7dc7408176f4d2
I20260925 10:12:06.539952 2317 raft_consensus.cc:686] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148 [term 1 LEADER]: Becoming Leader. State: Replica: e806fa1438b3421aaff4ce83def08148, State: Running, Role: LEADER
I20260925 10:12:06.540045 2317 consensus_queue.cc:237] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e806fa1438b3421aaff4ce83def08148" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39183 } }
I20260925 10:12:06.540125 2314 sys_catalog.cc:564] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:06.540378 2317 sys_catalog.cc:455] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e806fa1438b3421aaff4ce83def08148. Latest consensus state: current_term: 1 leader_uuid: "e806fa1438b3421aaff4ce83def08148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e806fa1438b3421aaff4ce83def08148" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39183 } } }
I20260925 10:12:06.540442 2317 sys_catalog.cc:458] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:06.540550 2318 sys_catalog.cc:455] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e806fa1438b3421aaff4ce83def08148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e806fa1438b3421aaff4ce83def08148" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39183 } } }
I20260925 10:12:06.540693 2323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:06.540693 2318 sys_catalog.cc:458] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:06.541342 2323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:06.543499 2323 catalog_manager.cc:1347] Generated new cluster ID: 81ae7d383a7a43b5954d7b8256cac857
I20260925 10:12:06.543565 2323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:12:06 dist-test-slave-s6tm krb5kdc[2233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:12:06.551591 2323 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:06.552368 2323 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:06.553936 2322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:06.540400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42955 (local address 127.0.0.1:39183)
0925 10:12:06.541674 (+ 1274us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:06.541686 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:06.541725 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:12:06.541866 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:06.541876 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:06.541965 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:06.542874 (+ 909us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:06.543169 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:06.544585 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:06.545712 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:06.546390 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:06.546490 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:06.549590 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:06.549596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:06.549596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:06.549614 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:06.551927 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:06.552537 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:06.552541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:06.552542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:06.552665 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:06.552957 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:06.552958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:06.552958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:06.553285 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:06.553447 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:06.553592 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 10:12:06.553696 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1210,"thread_start_us":69,"threads_started":1}
I20260925 10:12:06.554949 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:06.563048 2323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e806fa1438b3421aaff4ce83def08148: Generated new TSK 0
I20260925 10:12:06.563277 2323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:06.564364 2322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:06.556541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57049 (local address 127.0.0.1:39183)
0925 10:12:06.556582 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:06.556584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:06.556705 (+ 121us) server_negotiation.cc:407] Connection header received
0925 10:12:06.556754 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:06.556755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:06.556765 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:06.556837 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:06.557492 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:06.558363 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:06.559429 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:06.559767 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:06.559842 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:06.561365 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:06.561367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:06.561367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:06.561376 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:06.563130 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:06.563577 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:06.563578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:06.563578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:06.563629 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:06.563855 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:06.563855 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:06.563855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:06.564000 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:06.564104 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:06.564246 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 10:12:06.564302 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:12:06.566563 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 2246
W20260925 10:12:06.571220 2231 connection.cc:535] client connection to 127.0.0.1:39183 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39183): Connection reset by peer (error 104)
2026-09-25T10:12:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:06 dist-test-slave-s6tm krb5kdc[2345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:06 dist-test-slave-s6tm krb5kdc[2345](info): set up 2 sockets
Sep 25 10:12:06 dist-test-slave-s6tm krb5kdc[2345](info): commencing operation
krb5kdc: starting...
W20260925 10:12:07.609494 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:07 dist-test-slave-s6tm krb5kdc[2345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:12:07.735886 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37213
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:07.820429 2358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:07.820688 2358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:07.820734 2358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:07.821925 2358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:07.821969 2358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:07.821982 2358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:07.821995 2358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:07.822007 2358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:07.823475 2358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37213
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:07.823685 2358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:07.823880 2358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:12:07.829066 2358 server_base.cc:1034] running on GCE node
W20260925 10:12:07.828971 2364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:07.828977 2366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:07.828995 2363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:07.829481 2358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:07.829782 2358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:07.830945 2358 hybrid_clock.cc:648] HybridClock initialized: now 1790331127830931 us; error 45 us; skew 500 ppm
Sep 25 10:12:07 dist-test-slave-s6tm krb5kdc[2345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:07.833544 2358 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:12:07.834062 2358 webserver.cc:460] Webserver started at http://127.0.0.1:38469/ using document root <none> and password file <none>
I20260925 10:12:07.834321 2358 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:07.834389 2358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:07.834537 2358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:07.835645 2358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "71c2b31be3d744ef8b2565c2cc790d6f"
format_stamp: "Formatted at 2026-09-25 10:12:07 on dist-test-slave-s6tm"
server_key: "465e58e021d7dd60f73f3d95a8ea8f59"
server_key_iv: "7dc1d1a863b2b19f542809cbe1a61e9c"
server_key_version: "encryptionkey@0"
I20260925 10:12:07.836088 2358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "71c2b31be3d744ef8b2565c2cc790d6f"
format_stamp: "Formatted at 2026-09-25 10:12:07 on dist-test-slave-s6tm"
server_key: "465e58e021d7dd60f73f3d95a8ea8f59"
server_key_iv: "7dc1d1a863b2b19f542809cbe1a61e9c"
server_key_version: "encryptionkey@0"
I20260925 10:12:07.837492 2358 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:12:07.838397 2373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:07.838572 2358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:12:07.838671 2358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "71c2b31be3d744ef8b2565c2cc790d6f"
format_stamp: "Formatted at 2026-09-25 10:12:07 on dist-test-slave-s6tm"
server_key: "465e58e021d7dd60f73f3d95a8ea8f59"
server_key_iv: "7dc1d1a863b2b19f542809cbe1a61e9c"
server_key_version: "encryptionkey@0"
I20260925 10:12:07.838753 2358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:07.863533 2358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:07.865787 2358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:07.865958 2358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:07.870285 2358 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37213
I20260925 10:12:07.870342 2425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37213 every 8 connection(s)
I20260925 10:12:07.870724 2358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:07.871538 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 2358
I20260925 10:12:07.871677 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:07.871706 2426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:07.871887 14501 external_mini_cluster.cc:1428] Setting key 6c7472ca0bfdf74add1517bf82c0a573
I20260925 10:12:07.875510 2426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f: Bootstrap starting.
I20260925 10:12:07.876240 2426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:07.876541 2426 log.cc:824] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:07.877384 2426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f: No bootstrap required, opened a new log
I20260925 10:12:07.878144 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:07.872493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59875 (local address 127.0.0.1:37213)
0925 10:12:07.872652 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:07.872663 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:07.872697 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:12:07.873237 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:07.873246 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:07.873313 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:07.873471 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:07.873639 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:07.875034 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:07.875875 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:07.876529 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:07.876660 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:07.877451 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:07.877455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:07.877456 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:07.877661 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:07.877699 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:07.877713 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:07.877814 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:07.877890 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 10:12:07.877920 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260925 10:12:07.879220 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:07.879534 2426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c2b31be3d744ef8b2565c2cc790d6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37213 } }
I20260925 10:12:07.879678 2426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:07.879700 2426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c2b31be3d744ef8b2565c2cc790d6f, State: Initialized, Role: FOLLOWER
I20260925 10:12:07.879841 2426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c2b31be3d744ef8b2565c2cc790d6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37213 } }
I20260925 10:12:07.879913 2426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:07.879982 2426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:07.880033 2426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:07.880604 2426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c2b31be3d744ef8b2565c2cc790d6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37213 } }
I20260925 10:12:07.880712 2426 leader_election.cc:302] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71c2b31be3d744ef8b2565c2cc790d6f; no voters:
I20260925 10:12:07.880882 2426 leader_election.cc:288] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:07.880936 2436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:07.881100 2436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f [term 1 LEADER]: Becoming Leader. State: Replica: 71c2b31be3d744ef8b2565c2cc790d6f, State: Running, Role: LEADER
I20260925 10:12:07.881147 2426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:07.881215 2436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c2b31be3d744ef8b2565c2cc790d6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37213 } }
I20260925 10:12:07.881559 2436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71c2b31be3d744ef8b2565c2cc790d6f. Latest consensus state: current_term: 1 leader_uuid: "71c2b31be3d744ef8b2565c2cc790d6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c2b31be3d744ef8b2565c2cc790d6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37213 } } }
I20260925 10:12:07.881639 2436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:07.881570 2437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71c2b31be3d744ef8b2565c2cc790d6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c2b31be3d744ef8b2565c2cc790d6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37213 } } }
I20260925 10:12:07.881681 2437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:07.881975 2442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:07.882481 2442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:07.884248 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:07.880723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43431 (local address 127.0.0.1:37213)
0925 10:12:07.880767 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:07.880770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:07.880931 (+ 161us) server_negotiation.cc:407] Connection header received
0925 10:12:07.881076 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:07.881078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:07.881089 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:07.881135 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:07.881335 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:07.882011 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:07.883075 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:07.883294 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:07.883352 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:07.884033 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:07.884034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:07.884034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:07.884037 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:07.884050 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:07.884055 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:07.884116 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:07.884191 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 10:12:07.884201 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:12:07.889257 2442 catalog_manager.cc:1347] Generated new cluster ID: 9003248629bf4aa0bf0472ecd1b3bfd4
I20260925 10:12:07.889329 2442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:07.916220 2442 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:07.916894 2442 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:07.922048 2442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71c2b31be3d744ef8b2565c2cc790d6f: Generated new TSK 0
I20260925 10:12:07.922297 2442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:07.925024 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 2358
W20260925 10:12:07.930109 2343 connection.cc:535] client connection to 127.0.0.1:37213 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37213): Connection reset by peer (error 104)
2026-09-25T10:12:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:07 dist-test-slave-s6tm krb5kdc[2457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:07 dist-test-slave-s6tm krb5kdc[2457](info): set up 2 sockets
Sep 25 10:12:07 dist-test-slave-s6tm krb5kdc[2457](info): commencing operation
krb5kdc: starting...
W20260925 10:12:08.965922 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:09 dist-test-slave-s6tm krb5kdc[2457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:12:09.094610 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:09.178714 2470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:09.178972 2470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:09.179000 2470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:09.180328 2470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:09.180379 2470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:09.180397 2470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:09.180411 2470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:09.180424 2470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:09.182126 2470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33845
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:09.182350 2470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:09.182540 2470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:09.188196 2478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:09.188333 2470 server_base.cc:1034] running on GCE node
W20260925 10:12:09.188243 2475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:09.188457 2476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:09.188760 2470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:09.189134 2470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:09.190320 2470 hybrid_clock.cc:648] HybridClock initialized: now 1790331129190304 us; error 36 us; skew 500 ppm
Sep 25 10:12:09 dist-test-slave-s6tm krb5kdc[2457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:09.193186 2470 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:12:09.193686 2470 webserver.cc:460] Webserver started at http://127.0.0.1:35741/ using document root <none> and password file <none>
I20260925 10:12:09.193933 2470 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:09.193981 2470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:09.194263 2470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:09.195499 2470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "52fdda72fff249e7a67ffb5d9af7800e"
format_stamp: "Formatted at 2026-09-25 10:12:09 on dist-test-slave-s6tm"
server_key: "9dc09ed6acef6554b42601795d760b1f"
server_key_iv: "6c5a16b12525db7b5d188ca2b98c247b"
server_key_version: "encryptionkey@0"
I20260925 10:12:09.195906 2470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "52fdda72fff249e7a67ffb5d9af7800e"
format_stamp: "Formatted at 2026-09-25 10:12:09 on dist-test-slave-s6tm"
server_key: "9dc09ed6acef6554b42601795d760b1f"
server_key_iv: "6c5a16b12525db7b5d188ca2b98c247b"
server_key_version: "encryptionkey@0"
I20260925 10:12:09.197548 2470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:09.198532 2485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:09.198734 2470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:09.198853 2470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "52fdda72fff249e7a67ffb5d9af7800e"
format_stamp: "Formatted at 2026-09-25 10:12:09 on dist-test-slave-s6tm"
server_key: "9dc09ed6acef6554b42601795d760b1f"
server_key_iv: "6c5a16b12525db7b5d188ca2b98c247b"
server_key_version: "encryptionkey@0"
I20260925 10:12:09.198938 2470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:09.202953 2470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:09.205650 2470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:09.205817 2470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:09.209836 2470 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33845
I20260925 10:12:09.209887 2537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33845 every 8 connection(s)
I20260925 10:12:09.210253 2470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:09.211174 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:09.213675 2538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e: Bootstrap starting.
I20260925 10:12:09.214295 2538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:09.214620 2538 log.cc:824] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:09.215400 2538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e: No bootstrap required, opened a new log
I20260925 10:12:09.217159 2538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fdda72fff249e7a67ffb5d9af7800e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33845 } }
I20260925 10:12:09.217331 2538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:09.217394 2538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52fdda72fff249e7a67ffb5d9af7800e, State: Initialized, Role: FOLLOWER
I20260925 10:12:09.217556 2538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fdda72fff249e7a67ffb5d9af7800e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33845 } }
I20260925 10:12:09.217636 2538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:09.217681 2538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:09.217731 2538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:09.218294 2538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fdda72fff249e7a67ffb5d9af7800e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33845 } }
I20260925 10:12:09.218413 2538 leader_election.cc:302] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52fdda72fff249e7a67ffb5d9af7800e; no voters:
I20260925 10:12:09.218600 2538 leader_election.cc:288] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:09.218657 2541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:09.218871 2538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:09.218889 2541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e [term 1 LEADER]: Becoming Leader. State: Replica: 52fdda72fff249e7a67ffb5d9af7800e, State: Running, Role: LEADER
I20260925 10:12:09.219063 2541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fdda72fff249e7a67ffb5d9af7800e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33845 } }
I20260925 10:12:09.219586 2542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52fdda72fff249e7a67ffb5d9af7800e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fdda72fff249e7a67ffb5d9af7800e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33845 } } }
I20260925 10:12:09.219668 2542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:09.219766 2543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52fdda72fff249e7a67ffb5d9af7800e. Latest consensus state: current_term: 1 leader_uuid: "52fdda72fff249e7a67ffb5d9af7800e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52fdda72fff249e7a67ffb5d9af7800e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33845 } } }
I20260925 10:12:09.219838 2543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:09.220100 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 2470
I20260925 10:12:09.220216 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:09.220417 14501 external_mini_cluster.cc:1428] Setting key b7eab4fc86c54f7e9e0c2b53775c2135
W20260925 10:12:09.220801 2557 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:12:09.220888 2557 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:12:09.220952 2550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:09.221100 2550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:09.223196 2550 catalog_manager.cc:1347] Generated new cluster ID: 05f0cf49bed7481d8d58e50013b1bdbb
I20260925 10:12:09.223263 2550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:09.223511 2560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:09.221128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52953 (local address 127.0.0.1:33845)
0925 10:12:09.221309 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:09.221318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:09.221884 (+ 566us) server_negotiation.cc:407] Connection header received
0925 10:12:09.222032 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:09.222040 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:09.222074 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:09.222250 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:09.223028 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:09.223035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:09.223035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:09.223041 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:09.223088 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:09.223105 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:09.223184 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:09.223267 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 10:12:09.223316 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260925 10:12:09.224479 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:09.229228 2560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:09.226047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43811 (local address 127.0.0.1:33845)
0925 10:12:09.226082 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:09.226083 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:09.228185 (+ 2102us) server_negotiation.cc:407] Connection header received
0925 10:12:09.228263 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:09.228264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:09.228281 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:09.228348 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:09.228995 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:09.228996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:09.228996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:09.229000 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:09.229016 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:09.229020 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:09.229084 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:09.229141 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 10:12:09.229156 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:12:09.232117 2550 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:09.232290 2550 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:09.243748 2550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52fdda72fff249e7a67ffb5d9af7800e: Generated new TSK 0
I20260925 10:12:09.244230 2550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:09.245822 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 2470
2026-09-25T10:12:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:09 dist-test-slave-s6tm krb5kdc[2569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:09 dist-test-slave-s6tm krb5kdc[2569](info): set up 2 sockets
Sep 25 10:12:09 dist-test-slave-s6tm krb5kdc[2569](info): commencing operation
krb5kdc: starting...
W20260925 10:12:10.293486 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:10 dist-test-slave-s6tm krb5kdc[2569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:12:10.422798 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40705
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:10.505192 2582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:10.505419 2582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:10.505442 2582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:10.506547 2582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:10.506585 2582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:10.506599 2582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:10.506613 2582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:10.506624 2582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:10.508184 2582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:10.508409 2582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:10.508590 2582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:10.513620 2587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:10.513592 2590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:10.513595 2588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:10.513665 2582 server_base.cc:1034] running on GCE node
I20260925 10:12:10.514078 2582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:10.514389 2582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:10.515547 2582 hybrid_clock.cc:648] HybridClock initialized: now 1790331130515531 us; error 30 us; skew 500 ppm
Sep 25 10:12:10 dist-test-slave-s6tm krb5kdc[2569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:10.518324 2582 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:12:10.518774 2582 webserver.cc:460] Webserver started at http://127.0.0.1:33621/ using document root <none> and password file <none>
I20260925 10:12:10.519007 2582 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:10.519083 2582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:10.519222 2582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:10.520371 2582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "724dc709de014e8e82fde797c3764aba"
format_stamp: "Formatted at 2026-09-25 10:12:10 on dist-test-slave-s6tm"
server_key: "045e0b0e6361bf81b2b584bdb6f331dd"
server_key_iv: "8f3b3de31091b6430fed89d625c262a3"
server_key_version: "encryptionkey@0"
I20260925 10:12:10.520761 2582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "724dc709de014e8e82fde797c3764aba"
format_stamp: "Formatted at 2026-09-25 10:12:10 on dist-test-slave-s6tm"
server_key: "045e0b0e6361bf81b2b584bdb6f331dd"
server_key_iv: "8f3b3de31091b6430fed89d625c262a3"
server_key_version: "encryptionkey@0"
I20260925 10:12:10.522217 2582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:10.522972 2597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:10.523214 2582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:10.523332 2582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "724dc709de014e8e82fde797c3764aba"
format_stamp: "Formatted at 2026-09-25 10:12:10 on dist-test-slave-s6tm"
server_key: "045e0b0e6361bf81b2b584bdb6f331dd"
server_key_iv: "8f3b3de31091b6430fed89d625c262a3"
server_key_version: "encryptionkey@0"
I20260925 10:12:10.523411 2582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:10.534325 2582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:10.536589 2582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:10.536762 2582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:10.540858 2582 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40705
I20260925 10:12:10.540897 2649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40705 every 8 connection(s)
I20260925 10:12:10.541285 2582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:10.542318 2650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:10.545090 2650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba: Bootstrap starting.
I20260925 10:12:10.545729 2650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:10.546037 2650 log.cc:824] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:10.546834 2650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba: No bootstrap required, opened a new log
I20260925 10:12:10.547861 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 2582
I20260925 10:12:10.548041 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:10.548262 14501 external_mini_cluster.cc:1428] Setting key 2e742124494b95ab989fae979cd91bf7
I20260925 10:12:10.549219 2650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "724dc709de014e8e82fde797c3764aba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40705 } }
I20260925 10:12:10.549393 2650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:10.549463 2650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 724dc709de014e8e82fde797c3764aba, State: Initialized, Role: FOLLOWER
I20260925 10:12:10.549669 2650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "724dc709de014e8e82fde797c3764aba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40705 } }
I20260925 10:12:10.549772 2650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:10.549822 2650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:10.549878 2650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:10.550887 2650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "724dc709de014e8e82fde797c3764aba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40705 } }
I20260925 10:12:10.551016 2650 leader_election.cc:302] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 724dc709de014e8e82fde797c3764aba; no voters:
I20260925 10:12:10.551221 2650 leader_election.cc:288] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:10.551246 2655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:10.551369 2655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba [term 1 LEADER]: Becoming Leader. State: Replica: 724dc709de014e8e82fde797c3764aba, State: Running, Role: LEADER
I20260925 10:12:10.551522 2650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:10.551518 2655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "724dc709de014e8e82fde797c3764aba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40705 } }
I20260925 10:12:10.552237 2657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 724dc709de014e8e82fde797c3764aba. Latest consensus state: current_term: 1 leader_uuid: "724dc709de014e8e82fde797c3764aba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "724dc709de014e8e82fde797c3764aba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40705 } } }
I20260925 10:12:10.552335 2657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:10.552565 2656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "724dc709de014e8e82fde797c3764aba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "724dc709de014e8e82fde797c3764aba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40705 } } }
I20260925 10:12:10.552691 2656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:10.552829 2664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:10.553408 2664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 10:12:10 dist-test-slave-s6tm krb5kdc[2569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:12:10.555529 2664 catalog_manager.cc:1347] Generated new cluster ID: 716f3ec3583445898e264ac22f3791cc
I20260925 10:12:10.555586 2664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:10.560021 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:10.548855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36633 (local address 127.0.0.1:40705)
0925 10:12:10.549082 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:10.549091 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:10.549127 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:12:10.549235 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:10.549241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:10.549284 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:10.549423 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:10.550143 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:10.551064 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:10.552178 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:10.552950 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:10.553094 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:10.555882 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:10.555887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:10.555887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:10.555901 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:10.558177 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:10.558582 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:10.558586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:10.558586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:10.558734 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:10.558946 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:10.558947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:10.558947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:10.559310 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:10.559502 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:10.559579 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 10:12:10.559736 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":111,"threads_started":1}
I20260925 10:12:10.561369 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:10.565246 2664 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:10.565995 2664 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:10.569778 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:10.563104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49319 (local address 127.0.0.1:40705)
0925 10:12:10.563177 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:10.563179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:10.563189 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:12:10.563291 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:10.563292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:10.563302 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:10.563356 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:10.564081 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:10.564793 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:10.565698 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:10.565911 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:10.565986 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:10.567183 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:10.567185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:10.567185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:10.567192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:10.568661 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:10.569014 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:10.569015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:10.569015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:10.569072 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:10.569261 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:10.569262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:10.569262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:10.569371 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:10.569470 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:10.569571 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 10:12:10.569669 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:12:10.576942 2664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 724dc709de014e8e82fde797c3764aba: Generated new TSK 0
I20260925 10:12:10.577172 2664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:10.582533 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 2582
2026-09-25T10:12:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:10 dist-test-slave-s6tm krb5kdc[2681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:10 dist-test-slave-s6tm krb5kdc[2681](info): set up 2 sockets
Sep 25 10:12:10 dist-test-slave-s6tm krb5kdc[2681](info): commencing operation
krb5kdc: starting...
W20260925 10:12:11.617559 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:11 dist-test-slave-s6tm krb5kdc[2681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:12:11.744747 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34447
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:11.829224 2694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:11.829524 2694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:11.829555 2694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:11.830792 2694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:11.830842 2694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:11.830856 2694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:11.830868 2694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:11.830880 2694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:11.832515 2694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:11.832799 2694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:11.833081 2694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:11.838405 2700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:11.838399 2699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:11.838438 2694 server_base.cc:1034] running on GCE node
W20260925 10:12:11.838405 2702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:11.838903 2694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:11.839210 2694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:11.840359 2694 hybrid_clock.cc:648] HybridClock initialized: now 1790331131840336 us; error 34 us; skew 500 ppm
Sep 25 10:12:11 dist-test-slave-s6tm krb5kdc[2681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:11.849478 2694 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:12:11.850032 2694 webserver.cc:460] Webserver started at http://127.0.0.1:43215/ using document root <none> and password file <none>
I20260925 10:12:11.850284 2694 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:11.850334 2694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:11.850466 2694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:11.851557 2694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "fa35f3c2ce89471088ebde5942059faf"
format_stamp: "Formatted at 2026-09-25 10:12:11 on dist-test-slave-s6tm"
server_key: "68368ab5828120d2f2f1e0405dabc09f"
server_key_iv: "9de99a1de4c755d44fb04f0416b022e9"
server_key_version: "encryptionkey@0"
I20260925 10:12:11.851902 2694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "fa35f3c2ce89471088ebde5942059faf"
format_stamp: "Formatted at 2026-09-25 10:12:11 on dist-test-slave-s6tm"
server_key: "68368ab5828120d2f2f1e0405dabc09f"
server_key_iv: "9de99a1de4c755d44fb04f0416b022e9"
server_key_version: "encryptionkey@0"
I20260925 10:12:11.853355 2694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:11.854169 2709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:11.854422 2694 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:11.854503 2694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "fa35f3c2ce89471088ebde5942059faf"
format_stamp: "Formatted at 2026-09-25 10:12:11 on dist-test-slave-s6tm"
server_key: "68368ab5828120d2f2f1e0405dabc09f"
server_key_iv: "9de99a1de4c755d44fb04f0416b022e9"
server_key_version: "encryptionkey@0"
I20260925 10:12:11.854588 2694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:11.866603 2694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:11.869099 2694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:11.869292 2694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:11.873605 2694 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34447
I20260925 10:12:11.873638 2761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34447 every 8 connection(s)
I20260925 10:12:11.874028 2694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:11.874940 2762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:11.877344 2762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf: Bootstrap starting.
I20260925 10:12:11.877902 2762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:11.878139 2762 log.cc:824] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:11.878808 2762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf: No bootstrap required, opened a new log
I20260925 10:12:11.880208 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 2694
I20260925 10:12:11.880347 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:11.880549 14501 external_mini_cluster.cc:1428] Setting key 421ca09fa8ab0af8d8dbca6a7781eab5
I20260925 10:12:11.880590 2762 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa35f3c2ce89471088ebde5942059faf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34447 } }
I20260925 10:12:11.880793 2762 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:11.880846 2762 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa35f3c2ce89471088ebde5942059faf, State: Initialized, Role: FOLLOWER
I20260925 10:12:11.881016 2762 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa35f3c2ce89471088ebde5942059faf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34447 } }
I20260925 10:12:11.881150 2762 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:11.881201 2762 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:11.881304 2762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:11.882519 2762 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa35f3c2ce89471088ebde5942059faf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34447 } }
I20260925 10:12:11.882635 2762 leader_election.cc:302] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa35f3c2ce89471088ebde5942059faf; no voters:
I20260925 10:12:11.882851 2762 leader_election.cc:288] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:11.882956 2767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:11.883126 2767 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf [term 1 LEADER]: Becoming Leader. State: Replica: fa35f3c2ce89471088ebde5942059faf, State: Running, Role: LEADER
I20260925 10:12:11.883157 2762 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:11.883283 2767 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa35f3c2ce89471088ebde5942059faf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34447 } }
I20260925 10:12:11.885911 2768 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa35f3c2ce89471088ebde5942059faf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa35f3c2ce89471088ebde5942059faf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34447 } } }
I20260925 10:12:11.885996 2768 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:11.886255 2782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:11.886276 2769 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa35f3c2ce89471088ebde5942059faf. Latest consensus state: current_term: 1 leader_uuid: "fa35f3c2ce89471088ebde5942059faf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa35f3c2ce89471088ebde5942059faf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34447 } } }
I20260925 10:12:11.886570 2769 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:11.886744 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:11.881108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45447 (local address 127.0.0.1:34447)
0925 10:12:11.881306 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:11.881316 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:11.881356 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:12:11.881490 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:11.881498 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:11.881553 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:11.881715 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:11.881880 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:11.883325 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:11.884585 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:11.885018 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:11.885183 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:11.885866 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:11.885870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:11.885870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:11.886134 (+ 264us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:11.886177 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:11.886194 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:11.886270 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:11.886428 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 10:12:11.886496 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
I20260925 10:12:11.887065 2782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:11.887725 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:11.888767 2782 catalog_manager.cc:1347] Generated new cluster ID: eb21c4a63c8845fba7f4993391b9f225
I20260925 10:12:11.888867 2782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:11.895881 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:11.892287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60753 (local address 127.0.0.1:34447)
0925 10:12:11.892329 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:11.892332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:11.892473 (+ 141us) server_negotiation.cc:407] Connection header received
0925 10:12:11.892544 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:11.892545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:11.892556 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:11.892618 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:11.892781 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:11.893521 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:11.894578 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:11.894876 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:11.894933 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:11.895644 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:11.895646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:11.895646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:11.895651 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:11.895667 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:11.895671 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:11.895720 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:11.895774 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 10:12:11.895799 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:12:11.911258 2782 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:11.911854 2782 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:11.918851 2782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa35f3c2ce89471088ebde5942059faf: Generated new TSK 0
I20260925 10:12:11.919107 2782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:11.928786 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 2694
2026-09-25T10:12:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:11 dist-test-slave-s6tm krb5kdc[2793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:11 dist-test-slave-s6tm krb5kdc[2793](info): set up 2 sockets
Sep 25 10:12:11 dist-test-slave-s6tm krb5kdc[2793](info): commencing operation
krb5kdc: starting...
W20260925 10:12:13.001528 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:13 dist-test-slave-s6tm krb5kdc[2793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:12:13.124155 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40273
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:13.206688 2806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:13.206918 2806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:13.206943 2806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:13.208086 2806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:13.208124 2806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:13.208138 2806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:13.208151 2806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:13.208163 2806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:13.209714 2806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:13.209949 2806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:13.210150 2806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:13.215293 2812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:13.215313 2811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:13.215308 2814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:13.215672 2806 server_base.cc:1034] running on GCE node
I20260925 10:12:13.215975 2806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:13.216293 2806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:13.217448 2806 hybrid_clock.cc:648] HybridClock initialized: now 1790331133217432 us; error 33 us; skew 500 ppm
Sep 25 10:12:13 dist-test-slave-s6tm krb5kdc[2793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:13.220301 2806 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:12:13.220798 2806 webserver.cc:460] Webserver started at http://127.0.0.1:45897/ using document root <none> and password file <none>
I20260925 10:12:13.221026 2806 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:13.221102 2806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:13.221247 2806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:13.222348 2806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "d4a0a117b035499797f2978d90dc27e0"
format_stamp: "Formatted at 2026-09-25 10:12:13 on dist-test-slave-s6tm"
server_key: "1ba0e72c99886ed0d93d697e91097ecc"
server_key_iv: "383ff46c14b35f6e00714f8ec6d37c18"
server_key_version: "encryptionkey@0"
I20260925 10:12:13.222721 2806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "d4a0a117b035499797f2978d90dc27e0"
format_stamp: "Formatted at 2026-09-25 10:12:13 on dist-test-slave-s6tm"
server_key: "1ba0e72c99886ed0d93d697e91097ecc"
server_key_iv: "383ff46c14b35f6e00714f8ec6d37c18"
server_key_version: "encryptionkey@0"
I20260925 10:12:13.224201 2806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:12:13.224946 2821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:13.225193 2806 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:13.225268 2806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "d4a0a117b035499797f2978d90dc27e0"
format_stamp: "Formatted at 2026-09-25 10:12:13 on dist-test-slave-s6tm"
server_key: "1ba0e72c99886ed0d93d697e91097ecc"
server_key_iv: "383ff46c14b35f6e00714f8ec6d37c18"
server_key_version: "encryptionkey@0"
I20260925 10:12:13.225322 2806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:13.228943 2806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:13.230957 2806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:13.231076 2806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:13.235129 2806 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40273
I20260925 10:12:13.235162 2873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40273 every 8 connection(s)
I20260925 10:12:13.235533 2806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:13.236441 2874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:13.238951 2874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0: Bootstrap starting.
I20260925 10:12:13.239128 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 2806
I20260925 10:12:13.239260 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:13.239464 14501 external_mini_cluster.cc:1428] Setting key 318acd06b3a244faf3174354bb2354e6
I20260925 10:12:13.239713 2874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:13.240101 2874 log.cc:824] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:13.240849 2874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0: No bootstrap required, opened a new log
I20260925 10:12:13.242338 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:13.240245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52535 (local address 127.0.0.1:40273)
0925 10:12:13.240437 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:13.240448 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:13.240478 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:12:13.241067 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:13.241074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:13.241103 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:13.241225 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:13.241867 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:13.241870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:13.241870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:13.241874 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:13.241917 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:13.241927 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:13.241993 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:13.242079 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 10:12:13.242131 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
I20260925 10:12:13.243445 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:13.243680 2874 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a0a117b035499797f2978d90dc27e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } }
I20260925 10:12:13.243849 2874 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:13.243883 2874 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4a0a117b035499797f2978d90dc27e0, State: Initialized, Role: FOLLOWER
I20260925 10:12:13.244045 2874 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a0a117b035499797f2978d90dc27e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } }
I20260925 10:12:13.244146 2874 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:13.244177 2874 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:13.244221 2874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:13.245057 2874 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a0a117b035499797f2978d90dc27e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } }
I20260925 10:12:13.245183 2874 leader_election.cc:302] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4a0a117b035499797f2978d90dc27e0; no voters:
I20260925 10:12:13.245342 2874 leader_election.cc:288] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:13.245390 2885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:13.245527 2885 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0 [term 1 LEADER]: Becoming Leader. State: Replica: d4a0a117b035499797f2978d90dc27e0, State: Running, Role: LEADER
I20260925 10:12:13.245668 2874 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:13.245656 2885 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a0a117b035499797f2978d90dc27e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } }
I20260925 10:12:13.246008 2885 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a0a117b035499797f2978d90dc27e0. Latest consensus state: current_term: 1 leader_uuid: "d4a0a117b035499797f2978d90dc27e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a0a117b035499797f2978d90dc27e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } } }
I20260925 10:12:13.246099 2885 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:13.246152 2886 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4a0a117b035499797f2978d90dc27e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a0a117b035499797f2978d90dc27e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40273 } } }
I20260925 10:12:13.246274 2886 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:13.246318 2890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:13.246694 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:13.245148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53097 (local address 127.0.0.1:40273)
0925 10:12:13.245189 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:13.245191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:13.245330 (+ 139us) server_negotiation.cc:407] Connection header received
0925 10:12:13.245399 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:13.245400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:13.245406 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:13.245459 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:13.246376 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:13.246377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:13.246377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:13.246382 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:13.246396 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:13.246401 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:13.246458 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:13.246609 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 10:12:13.246625 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:12:13.246910 2890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:13.249056 2890 catalog_manager.cc:1347] Generated new cluster ID: 2fefbcf0ebb94852a1350cdb38888f48
I20260925 10:12:13.249115 2890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:13.257586 2890 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:13.257776 2890 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:13.273804 2890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4a0a117b035499797f2978d90dc27e0: Generated new TSK 0
I20260925 10:12:13.274060 2890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:13.281183 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 2806
2026-09-25T10:12:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:13 dist-test-slave-s6tm krb5kdc[2906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:13 dist-test-slave-s6tm krb5kdc[2906](info): set up 2 sockets
Sep 25 10:12:13 dist-test-slave-s6tm krb5kdc[2906](info): commencing operation
krb5kdc: starting...
W20260925 10:12:14.321480 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:14 dist-test-slave-s6tm krb5kdc[2906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:12:14.444394 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45571
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:14.527588 2919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:14.527807 2919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:14.527830 2919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:14.529129 2919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:14.529181 2919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:14.529196 2919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:14.529209 2919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:14.529222 2919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:14.530758 2919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:14.530974 2919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:14.531152 2919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:14.536552 2927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:14.536760 2919 server_base.cc:1034] running on GCE node
W20260925 10:12:14.536543 2925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:14.536520 2924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:14.537284 2919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:14.537622 2919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:14.538794 2919 hybrid_clock.cc:648] HybridClock initialized: now 1790331134538772 us; error 41 us; skew 500 ppm
Sep 25 10:12:14 dist-test-slave-s6tm krb5kdc[2906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:14.541417 2919 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:12:14.541941 2919 webserver.cc:460] Webserver started at http://127.0.0.1:35793/ using document root <none> and password file <none>
I20260925 10:12:14.542181 2919 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:14.542258 2919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:14.542423 2919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:14.543557 2919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "8b1ba2a3766f4d338e91ab0037847aae"
format_stamp: "Formatted at 2026-09-25 10:12:14 on dist-test-slave-s6tm"
server_key: "40bd9d8c5d51e27b884ad026a9faead3"
server_key_iv: "6e6d814c129a3acc4c0b473eb746d34c"
server_key_version: "encryptionkey@0"
I20260925 10:12:14.544004 2919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "8b1ba2a3766f4d338e91ab0037847aae"
format_stamp: "Formatted at 2026-09-25 10:12:14 on dist-test-slave-s6tm"
server_key: "40bd9d8c5d51e27b884ad026a9faead3"
server_key_iv: "6e6d814c129a3acc4c0b473eb746d34c"
server_key_version: "encryptionkey@0"
I20260925 10:12:14.545921 2919 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 10:12:14.546767 2934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:14.546955 2919 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:14.547061 2919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "8b1ba2a3766f4d338e91ab0037847aae"
format_stamp: "Formatted at 2026-09-25 10:12:14 on dist-test-slave-s6tm"
server_key: "40bd9d8c5d51e27b884ad026a9faead3"
server_key_iv: "6e6d814c129a3acc4c0b473eb746d34c"
server_key_version: "encryptionkey@0"
I20260925 10:12:14.547154 2919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:14.555774 2919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:14.557862 2919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:14.558034 2919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:14.562330 2919 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45571
I20260925 10:12:14.562366 2986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45571 every 8 connection(s)
I20260925 10:12:14.562731 2919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:14.563769 2987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:14.566246 2987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae: Bootstrap starting.
I20260925 10:12:14.566879 2987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:14.567206 2987 log.cc:824] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:14.568020 2987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae: No bootstrap required, opened a new log
I20260925 10:12:14.569487 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 2919
I20260925 10:12:14.569630 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:14.569835 14501 external_mini_cluster.cc:1428] Setting key 6a97b7a6777bc851a260fa0c83d0c0f9
I20260925 10:12:14.569857 2987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1ba2a3766f4d338e91ab0037847aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45571 } }
I20260925 10:12:14.570060 2987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:14.570118 2987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b1ba2a3766f4d338e91ab0037847aae, State: Initialized, Role: FOLLOWER
I20260925 10:12:14.570268 2987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1ba2a3766f4d338e91ab0037847aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45571 } }
I20260925 10:12:14.570384 2987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:14.570443 2987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:14.570497 2987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:14.571637 2987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1ba2a3766f4d338e91ab0037847aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45571 } }
I20260925 10:12:14.571748 2987 leader_election.cc:302] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b1ba2a3766f4d338e91ab0037847aae; no voters:
I20260925 10:12:14.571899 2987 leader_election.cc:288] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:14.571990 2992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:14.572155 2992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae [term 1 LEADER]: Becoming Leader. State: Replica: 8b1ba2a3766f4d338e91ab0037847aae, State: Running, Role: LEADER
I20260925 10:12:14.572219 2987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:14.572301 2992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1ba2a3766f4d338e91ab0037847aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45571 } }
I20260925 10:12:14.572657 2993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b1ba2a3766f4d338e91ab0037847aae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1ba2a3766f4d338e91ab0037847aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45571 } } }
I20260925 10:12:14.572644 2992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b1ba2a3766f4d338e91ab0037847aae. Latest consensus state: current_term: 1 leader_uuid: "8b1ba2a3766f4d338e91ab0037847aae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b1ba2a3766f4d338e91ab0037847aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45571 } } }
I20260925 10:12:14.572835 2992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:14.572825 2993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae [sys.catalog]: This master's current role is: LEADER
W20260925 10:12:14.574102 3008 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:12:14.574154 3008 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:12:14.574198 3001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:14.574345 3001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:14.575969 3001 catalog_manager.cc:1347] Generated new cluster ID: 8fbd115d030641c68e9fe7ae6060e218
I20260925 10:12:14.576025 3001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:12:14 dist-test-slave-s6tm krb5kdc[2906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:12:14.582607 2991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:14.570388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:43693 (local address 127.0.0.1:45571)
0925 10:12:14.570595 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:14.570604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:14.570641 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:12:14.570770 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:14.570777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:14.570835 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:14.571007 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:14.571604 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:14.573127 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:14.574191 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:14.574713 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:14.574833 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:14.577876 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:14.577880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:14.577881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:14.577892 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:14.579919 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:14.580419 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:14.580423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:14.580423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:14.580558 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:14.580823 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:14.580824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:14.580824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:14.581141 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:14.581821 (+ 680us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:14.582273 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 10:12:14.582383 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":146,"thread_start_us":112,"threads_started":1}
I20260925 10:12:14.582754 3001 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:14.583417 3001 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:14.583557 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:14.591446 3001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b1ba2a3766f4d338e91ab0037847aae: Generated new TSK 0
I20260925 10:12:14.591640 3001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:14.592489 2991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:14.585115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:48017 (local address 127.0.0.1:45571)
0925 10:12:14.585149 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:14.585151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:14.585220 (+ 69us) server_negotiation.cc:407] Connection header received
0925 10:12:14.585279 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:14.585279 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:14.585288 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:14.585354 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:14.585954 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:14.586751 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:14.587750 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:14.588103 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:14.588174 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:14.589531 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:14.589533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:14.589534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:14.589540 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:14.591135 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:14.591663 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:14.591665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:14.591665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:14.591725 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:14.591971 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:14.591971 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:14.591972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:14.592113 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:14.592216 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:14.592332 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 10:12:14.592397 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 10:12:14.594509 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 2919
W20260925 10:12:14.598757 2904 connection.cc:535] client connection to 127.0.0.1:45571 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45571): Connection reset by peer (error 104)
2026-09-25T10:12:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:14 dist-test-slave-s6tm krb5kdc[3018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:14 dist-test-slave-s6tm krb5kdc[3018](info): set up 2 sockets
Sep 25 10:12:14 dist-test-slave-s6tm krb5kdc[3018](info): commencing operation
krb5kdc: starting...
W20260925 10:12:15.645578 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:15 dist-test-slave-s6tm krb5kdc[3018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:12:15.771886 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38319
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:15.855662 3031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:15.855921 3031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:15.855984 3031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:15.857187 3031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:15.857271 3031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:15.857299 3031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:15.857329 3031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:15.857354 3031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:15.858855 3031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:15.859134 3031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:15.859375 3031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:15.864437 3036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:15.864639 3039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:15.864436 3037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:15.864769 3031 server_base.cc:1034] running on GCE node
I20260925 10:12:15.865020 3031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:15.865326 3031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:15.866528 3031 hybrid_clock.cc:648] HybridClock initialized: now 1790331135866515 us; error 45 us; skew 500 ppm
Sep 25 10:12:15 dist-test-slave-s6tm krb5kdc[3018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:15.869184 3031 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:12:15.869733 3031 webserver.cc:460] Webserver started at http://127.0.0.1:37103/ using document root <none> and password file <none>
I20260925 10:12:15.869982 3031 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:15.870052 3031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:15.870210 3031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:15.871320 3031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "085140e2edcb434088c6fa8b0644336d"
format_stamp: "Formatted at 2026-09-25 10:12:15 on dist-test-slave-s6tm"
server_key: "8582428894b26934d4eb5c61337cc582"
server_key_iv: "8b4ae379c36bd390f7e9cdb1366f54d2"
server_key_version: "encryptionkey@0"
I20260925 10:12:15.871697 3031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "085140e2edcb434088c6fa8b0644336d"
format_stamp: "Formatted at 2026-09-25 10:12:15 on dist-test-slave-s6tm"
server_key: "8582428894b26934d4eb5c61337cc582"
server_key_iv: "8b4ae379c36bd390f7e9cdb1366f54d2"
server_key_version: "encryptionkey@0"
I20260925 10:12:15.873176 3031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:12:15.873986 3046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:15.874161 3031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:12:15.874311 3031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "085140e2edcb434088c6fa8b0644336d"
format_stamp: "Formatted at 2026-09-25 10:12:15 on dist-test-slave-s6tm"
server_key: "8582428894b26934d4eb5c61337cc582"
server_key_iv: "8b4ae379c36bd390f7e9cdb1366f54d2"
server_key_version: "encryptionkey@0"
I20260925 10:12:15.874398 3031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:15.887029 3031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:15.889110 3031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:15.889303 3031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:15.893919 3031 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38319
I20260925 10:12:15.893955 3098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38319 every 8 connection(s)
I20260925 10:12:15.894317 3031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:15.895179 3099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:15.897069 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 3031
I20260925 10:12:15.897217 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:15.897406 14501 external_mini_cluster.cc:1428] Setting key afa868a2be98431efec1764b1956efa8
I20260925 10:12:15.897827 3099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d: Bootstrap starting.
I20260925 10:12:15.898566 3099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:15.898902 3099 log.cc:824] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:15.899629 3099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d: No bootstrap required, opened a new log
I20260925 10:12:15.901456 3099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085140e2edcb434088c6fa8b0644336d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38319 } }
I20260925 10:12:15.901615 3099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:15.901664 3099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085140e2edcb434088c6fa8b0644336d, State: Initialized, Role: FOLLOWER
I20260925 10:12:15.901818 3099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085140e2edcb434088c6fa8b0644336d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38319 } }
I20260925 10:12:15.901912 3099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:15.901950 3099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:15.902004 3099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:15.902549 3099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085140e2edcb434088c6fa8b0644336d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38319 } }
I20260925 10:12:15.902666 3099 leader_election.cc:302] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 085140e2edcb434088c6fa8b0644336d; no voters:
I20260925 10:12:15.902841 3099 leader_election.cc:288] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:15.902868 3104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:15.903007 3104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d [term 1 LEADER]: Becoming Leader. State: Replica: 085140e2edcb434088c6fa8b0644336d, State: Running, Role: LEADER
I20260925 10:12:15.903095 3104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085140e2edcb434088c6fa8b0644336d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38319 } }
I20260925 10:12:15.903112 3099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:15.903429 3102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:15.897939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:47333 (local address 127.0.0.1:38319)
0925 10:12:15.898148 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:15.898159 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:15.898199 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:12:15.898822 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:15.898833 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:15.898905 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:15.899060 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:15.899385 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:15.900431 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:15.901246 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:15.901695 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:15.901804 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:15.902556 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:15.902561 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:15.902561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:15.902787 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:15.902828 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:15.902845 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:15.902985 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:15.903186 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 10:12:15.903219 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":86,"threads_started":1}
I20260925 10:12:15.903407 3104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 085140e2edcb434088c6fa8b0644336d. Latest consensus state: current_term: 1 leader_uuid: "085140e2edcb434088c6fa8b0644336d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085140e2edcb434088c6fa8b0644336d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38319 } } }
I20260925 10:12:15.903527 3104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:15.903419 3105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "085140e2edcb434088c6fa8b0644336d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085140e2edcb434088c6fa8b0644336d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38319 } } }
I20260925 10:12:15.903724 3105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:15.903836 3109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:15.904368 3109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:15.904495 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:15.906199 3109 catalog_manager.cc:1347] Generated new cluster ID: 3e2e1fe2ea7d4cc88794fd07158df389
I20260925 10:12:15.906297 3109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:15.909754 3102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:15.906106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:52423 (local address 127.0.0.1:38319)
0925 10:12:15.906139 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:15.906141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:15.906274 (+ 133us) server_negotiation.cc:407] Connection header received
0925 10:12:15.906323 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:15.906324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:15.906335 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:15.906390 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:15.906581 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:15.907297 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:15.908360 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:15.908677 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:15.908737 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:15.909476 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:15.909477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:15.909478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:15.909482 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:15.909497 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:15.909501 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:15.909556 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:15.909672 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 10:12:15.909683 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 10:12:15.915915 3109 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:15.916517 3109 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:15.920969 3109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 085140e2edcb434088c6fa8b0644336d: Generated new TSK 0
I20260925 10:12:15.921166 3109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:15.923411 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 3031
W20260925 10:12:15.927899 3016 connection.cc:535] client connection to 127.0.0.1:38319 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38319): Connection reset by peer (error 104)
2026-09-25T10:12:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:15 dist-test-slave-s6tm krb5kdc[3130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:15 dist-test-slave-s6tm krb5kdc[3130](info): set up 2 sockets
Sep 25 10:12:15 dist-test-slave-s6tm krb5kdc[3130](info): commencing operation
krb5kdc: starting...
W20260925 10:12:16.973558 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:17 dist-test-slave-s6tm krb5kdc[3130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:12:17.099668 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46413
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:17.182476 3143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:17.182740 3143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:17.182780 3143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:17.184033 3143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:17.184098 3143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:17.184122 3143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:17.184149 3143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:17.184180 3143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:17.185695 3143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:17.185984 3143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:17.186234 3143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:17.191270 3151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:17.191339 3143 server_base.cc:1034] running on GCE node
W20260925 10:12:17.191278 3149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:17.191303 3148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:17.192010 3143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:17.192272 3143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:17.193427 3143 hybrid_clock.cc:648] HybridClock initialized: now 1790331137193414 us; error 31 us; skew 500 ppm
Sep 25 10:12:17 dist-test-slave-s6tm krb5kdc[3130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:17.205304 3143 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:12:17.205906 3143 webserver.cc:460] Webserver started at http://127.0.0.1:37195/ using document root <none> and password file <none>
I20260925 10:12:17.206179 3143 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:17.206228 3143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:17.206404 3143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:17.207602 3143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "a21e2411dd94457ea2c53e41a5581efc"
format_stamp: "Formatted at 2026-09-25 10:12:17 on dist-test-slave-s6tm"
server_key: "c5b8cb1052505aca015ef24ddf19f28f"
server_key_iv: "4dcd97ffceab1ee011c5248a3bbfb697"
server_key_version: "encryptionkey@0"
I20260925 10:12:17.208081 3143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "a21e2411dd94457ea2c53e41a5581efc"
format_stamp: "Formatted at 2026-09-25 10:12:17 on dist-test-slave-s6tm"
server_key: "c5b8cb1052505aca015ef24ddf19f28f"
server_key_iv: "4dcd97ffceab1ee011c5248a3bbfb697"
server_key_version: "encryptionkey@0"
I20260925 10:12:17.209566 3143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:12:17.210476 3158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:17.210681 3143 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:12:17.210811 3143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "a21e2411dd94457ea2c53e41a5581efc"
format_stamp: "Formatted at 2026-09-25 10:12:17 on dist-test-slave-s6tm"
server_key: "c5b8cb1052505aca015ef24ddf19f28f"
server_key_iv: "4dcd97ffceab1ee011c5248a3bbfb697"
server_key_version: "encryptionkey@0"
I20260925 10:12:17.210906 3143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:17.214721 3143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:17.216987 3143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:17.217162 3143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:17.221321 3143 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46413
I20260925 10:12:17.221374 3210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46413 every 8 connection(s)
I20260925 10:12:17.221753 3143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:17.222661 3211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:17.224757 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 3143
I20260925 10:12:17.224887 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:17.225081 14501 external_mini_cluster.cc:1428] Setting key ef92e13a787a70e02b74d867f533d8a5
I20260925 10:12:17.225550 3211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc: Bootstrap starting.
I20260925 10:12:17.226249 3211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:17.226619 3211 log.cc:824] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:17.227495 3211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc: No bootstrap required, opened a new log
I20260925 10:12:17.227767 3214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:17.225657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:38363 (local address 127.0.0.1:46413)
0925 10:12:17.225858 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:17.225866 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:17.225892 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:12:17.226323 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:17.226330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:17.226387 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:17.226526 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:17.227319 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:17.227330 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:17.227330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:17.227334 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:17.227374 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:17.227384 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:17.227457 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:17.227530 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 10:12:17.227597 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260925 10:12:17.229024 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:17.230330 3211 raft_consensus.cc:348] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21e2411dd94457ea2c53e41a5581efc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46413 } }
I20260925 10:12:17.230537 3211 raft_consensus.cc:374] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:17.230578 3211 raft_consensus.cc:729] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a21e2411dd94457ea2c53e41a5581efc, State: Initialized, Role: FOLLOWER
I20260925 10:12:17.230719 3211 consensus_queue.cc:260] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21e2411dd94457ea2c53e41a5581efc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46413 } }
I20260925 10:12:17.230826 3211 raft_consensus.cc:388] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:17.230890 3211 raft_consensus.cc:482] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:17.230947 3211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:17.231572 3211 raft_consensus.cc:504] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21e2411dd94457ea2c53e41a5581efc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46413 } }
I20260925 10:12:17.231699 3211 leader_election.cc:302] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a21e2411dd94457ea2c53e41a5581efc; no voters:
I20260925 10:12:17.231900 3211 leader_election.cc:288] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:17.232061 3214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:17.230764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:58049 (local address 127.0.0.1:46413)
0925 10:12:17.230808 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:17.230810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:17.230946 (+ 136us) server_negotiation.cc:407] Connection header received
0925 10:12:17.231046 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:17.231047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:17.231052 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:17.231106 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:17.231771 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:17.231773 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:17.231773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:17.231778 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:17.231792 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:17.231797 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:17.231848 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:17.231950 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 10:12:17.231976 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:12:17.232285 3211 sys_catalog.cc:564] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:17.231945 3221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:17.233485 3221 raft_consensus.cc:686] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc [term 1 LEADER]: Becoming Leader. State: Replica: a21e2411dd94457ea2c53e41a5581efc, State: Running, Role: LEADER
I20260925 10:12:17.233618 3221 consensus_queue.cc:237] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21e2411dd94457ea2c53e41a5581efc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46413 } }
I20260925 10:12:17.234130 3233 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a21e2411dd94457ea2c53e41a5581efc. Latest consensus state: current_term: 1 leader_uuid: "a21e2411dd94457ea2c53e41a5581efc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21e2411dd94457ea2c53e41a5581efc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46413 } } }
I20260925 10:12:17.234452 3233 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:17.234315 3222 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a21e2411dd94457ea2c53e41a5581efc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21e2411dd94457ea2c53e41a5581efc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46413 } } }
I20260925 10:12:17.234570 3222 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc [sys.catalog]: This master's current role is: LEADER
W20260925 10:12:17.234750 3235 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:12:17.234822 3235 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:12:17.235000 3237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:17.235168 3237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:17.236812 3237 catalog_manager.cc:1347] Generated new cluster ID: 73c09fea9e9c4b499d3b2a63e00cd193
I20260925 10:12:17.236903 3237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:17.255777 3237 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:17.256120 3237 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:17.262265 3237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a21e2411dd94457ea2c53e41a5581efc: Generated new TSK 0
I20260925 10:12:17.262480 3237 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:17.266846 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 3143
2026-09-25T10:12:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:17 dist-test-slave-s6tm krb5kdc[3242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:17 dist-test-slave-s6tm krb5kdc[3242](info): set up 2 sockets
Sep 25 10:12:17 dist-test-slave-s6tm krb5kdc[3242](info): commencing operation
krb5kdc: starting...
W20260925 10:12:18.377568 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:18 dist-test-slave-s6tm krb5kdc[3242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:12:18.509168 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43349
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:18.593179 3255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:18.593405 3255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:18.593428 3255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:18.594738 3255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:18.594789 3255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:18.594803 3255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:18.594816 3255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:18.594830 3255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:18.596371 3255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43349
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:18.596634 3255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:18.596882 3255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:12:18.602119 3255 server_base.cc:1034] running on GCE node
W20260925 10:12:18.602059 3260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:18.602059 3263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:18.602059 3261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:18.602594 3255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:18.602912 3255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:18.604075 3255 hybrid_clock.cc:648] HybridClock initialized: now 1790331138604051 us; error 42 us; skew 500 ppm
Sep 25 10:12:18 dist-test-slave-s6tm krb5kdc[3242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:18.606699 3255 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:12:18.607200 3255 webserver.cc:460] Webserver started at http://127.0.0.1:43955/ using document root <none> and password file <none>
I20260925 10:12:18.607457 3255 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:18.607529 3255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:18.607682 3255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:18.608856 3255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "ccf1212337804bd4b5f1686ca5d3e452"
format_stamp: "Formatted at 2026-09-25 10:12:18 on dist-test-slave-s6tm"
server_key: "2e224196ef0d1906e16ab63b4642ff9d"
server_key_iv: "180ebf9bc8f784bf9ad906a2e02739de"
server_key_version: "encryptionkey@0"
I20260925 10:12:18.609346 3255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "ccf1212337804bd4b5f1686ca5d3e452"
format_stamp: "Formatted at 2026-09-25 10:12:18 on dist-test-slave-s6tm"
server_key: "2e224196ef0d1906e16ab63b4642ff9d"
server_key_iv: "180ebf9bc8f784bf9ad906a2e02739de"
server_key_version: "encryptionkey@0"
I20260925 10:12:18.610777 3255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:12:18.611578 3270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:18.611806 3255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:18.611914 3255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "ccf1212337804bd4b5f1686ca5d3e452"
format_stamp: "Formatted at 2026-09-25 10:12:18 on dist-test-slave-s6tm"
server_key: "2e224196ef0d1906e16ab63b4642ff9d"
server_key_iv: "180ebf9bc8f784bf9ad906a2e02739de"
server_key_version: "encryptionkey@0"
I20260925 10:12:18.612061 3255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:18.624578 3255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:18.626937 3255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:18.627107 3255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:18.631440 3255 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43349
I20260925 10:12:18.631479 3322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43349 every 8 connection(s)
I20260925 10:12:18.631872 3255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:18.632798 3323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:18.634665 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 3255
I20260925 10:12:18.634845 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:18.635138 14501 external_mini_cluster.cc:1428] Setting key 04086bbcc527332ccb409c116c68d5b7
I20260925 10:12:18.635335 3323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452: Bootstrap starting.
I20260925 10:12:18.636152 3323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:18.636490 3323 log.cc:824] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:18.637305 3323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452: No bootstrap required, opened a new log
I20260925 10:12:18.639395 3323 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf1212337804bd4b5f1686ca5d3e452" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43349 } }
I20260925 10:12:18.639587 3323 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:18.639642 3323 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf1212337804bd4b5f1686ca5d3e452, State: Initialized, Role: FOLLOWER
I20260925 10:12:18.639789 3323 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf1212337804bd4b5f1686ca5d3e452" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43349 } }
I20260925 10:12:18.639901 3323 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:18.639999 3323 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:18.640062 3323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:18.640843 3323 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf1212337804bd4b5f1686ca5d3e452" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43349 } }
I20260925 10:12:18.640990 3323 leader_election.cc:302] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccf1212337804bd4b5f1686ca5d3e452; no voters:
I20260925 10:12:18.641184 3323 leader_election.cc:288] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:18.641247 3328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:18.641387 3328 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452 [term 1 LEADER]: Becoming Leader. State: Replica: ccf1212337804bd4b5f1686ca5d3e452, State: Running, Role: LEADER
I20260925 10:12:18.641474 3323 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:18.641556 3328 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf1212337804bd4b5f1686ca5d3e452" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43349 } }
I20260925 10:12:18.641896 3328 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccf1212337804bd4b5f1686ca5d3e452. Latest consensus state: current_term: 1 leader_uuid: "ccf1212337804bd4b5f1686ca5d3e452" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf1212337804bd4b5f1686ca5d3e452" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43349 } } }
I20260925 10:12:18.641960 3328 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:18.642223 3334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 10:12:18 dist-test-slave-s6tm krb5kdc[3242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:12:18.642683 3334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:18.643803 3329 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccf1212337804bd4b5f1686ca5d3e452" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf1212337804bd4b5f1686ca5d3e452" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43349 } } }
I20260925 10:12:18.643962 3329 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:18.644395 3334 catalog_manager.cc:1347] Generated new cluster ID: 7ff8ec71cc754bcdaff431083136b65f
I20260925 10:12:18.644451 3334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:18.647315 3326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:18.635702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:50827 (local address 127.0.0.1:43349)
0925 10:12:18.635919 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:18.635953 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:18.635989 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:12:18.636612 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:18.636621 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:18.636692 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:18.636866 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:18.637614 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:18.638524 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:18.639300 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:18.639805 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:18.639924 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:18.643091 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:18.643096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:18.643096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:18.643111 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:18.645510 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:18.645971 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:18.645974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:18.645974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:18.646080 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:18.646342 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:18.646344 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:18.646344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:18.646622 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:18.646819 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:18.646945 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 10:12:18.647060 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":112,"threads_started":1}
I20260925 10:12:18.648541 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:18.657522 3326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:18.650106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:57977 (local address 127.0.0.1:43349)
0925 10:12:18.650150 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:18.650152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:18.650281 (+ 129us) server_negotiation.cc:407] Connection header received
0925 10:12:18.650364 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:18.650365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:18.650376 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:18.650440 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:18.651047 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:18.651622 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:18.652585 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:18.652854 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:18.652950 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:18.654382 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:18.654383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:18.654384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:18.654392 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:18.656136 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:18.656567 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:18.656568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:18.656569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:18.656640 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:18.656895 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:18.656896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:18.656897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:18.657051 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:18.657195 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:18.657321 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 10:12:18.657390 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:12:18.669142 3334 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:18.669708 3334 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:18.688529 3334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccf1212337804bd4b5f1686ca5d3e452: Generated new TSK 0
I20260925 10:12:18.688771 3334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:18.695811 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 3255
W20260925 10:12:18.700369 3240 connection.cc:535] client connection to 127.0.0.1:43349 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43349): Connection reset by peer (error 104)
2026-09-25T10:12:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:18 dist-test-slave-s6tm krb5kdc[3354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:18 dist-test-slave-s6tm krb5kdc[3354](info): set up 2 sockets
Sep 25 10:12:18 dist-test-slave-s6tm krb5kdc[3354](info): commencing operation
krb5kdc: starting...
W20260925 10:12:19.765506 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:19 dist-test-slave-s6tm krb5kdc[3354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:12:19.891885 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:19.975056 3367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:19.975277 3367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:19.975301 3367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:19.976558 3367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:19.976609 3367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:19.976624 3367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:19.976637 3367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:19.976691 3367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:19.978288 3367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42271
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:19.978585 3367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:19.978871 3367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:12:19.984098 3367 server_base.cc:1034] running on GCE node
W20260925 10:12:19.984115 3373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:19.984138 3372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:19.984265 3375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:19.984519 3367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:19.984843 3367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:19.985985 3367 hybrid_clock.cc:648] HybridClock initialized: now 1790331139985971 us; error 38 us; skew 500 ppm
Sep 25 10:12:19 dist-test-slave-s6tm krb5kdc[3354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:19.988595 3367 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:12:19.989007 3367 webserver.cc:460] Webserver started at http://127.0.0.1:44345/ using document root <none> and password file <none>
I20260925 10:12:19.989202 3367 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:19.989245 3367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:19.989339 3367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:19.990388 3367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "daa4cd3592af4eeeb38b2040fd892da9"
format_stamp: "Formatted at 2026-09-25 10:12:19 on dist-test-slave-s6tm"
server_key: "9cfd85ada9c6abbd85dcbe5a500bc606"
server_key_iv: "0400f54e58eb80d6aaf94461354af64e"
server_key_version: "encryptionkey@0"
I20260925 10:12:19.990708 3367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "daa4cd3592af4eeeb38b2040fd892da9"
format_stamp: "Formatted at 2026-09-25 10:12:19 on dist-test-slave-s6tm"
server_key: "9cfd85ada9c6abbd85dcbe5a500bc606"
server_key_iv: "0400f54e58eb80d6aaf94461354af64e"
server_key_version: "encryptionkey@0"
I20260925 10:12:19.992079 3367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:12:19.992866 3382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:19.993069 3367 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:19.993178 3367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "daa4cd3592af4eeeb38b2040fd892da9"
format_stamp: "Formatted at 2026-09-25 10:12:19 on dist-test-slave-s6tm"
server_key: "9cfd85ada9c6abbd85dcbe5a500bc606"
server_key_iv: "0400f54e58eb80d6aaf94461354af64e"
server_key_version: "encryptionkey@0"
I20260925 10:12:19.993269 3367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:20.005908 3367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:20.008293 3367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:20.008469 3367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:20.012351 3367 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42271
I20260925 10:12:20.012390 3434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42271 every 8 connection(s)
I20260925 10:12:20.012754 3367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:20.013712 3435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:20.016088 3435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9: Bootstrap starting.
I20260925 10:12:20.016619 3435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:20.016857 3435 log.cc:824] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:20.017515 3435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9: No bootstrap required, opened a new log
I20260925 10:12:20.017589 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 3367
I20260925 10:12:20.017745 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:20.017922 14501 external_mini_cluster.cc:1428] Setting key b6d7af8783ec8197aff694707a21ec2c
I20260925 10:12:20.019507 3435 raft_consensus.cc:348] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa4cd3592af4eeeb38b2040fd892da9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42271 } }
I20260925 10:12:20.019690 3435 raft_consensus.cc:374] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:20.019748 3435 raft_consensus.cc:729] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa4cd3592af4eeeb38b2040fd892da9, State: Initialized, Role: FOLLOWER
I20260925 10:12:20.019896 3435 consensus_queue.cc:260] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa4cd3592af4eeeb38b2040fd892da9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42271 } }
I20260925 10:12:20.020018 3435 raft_consensus.cc:388] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:20.020061 3435 raft_consensus.cc:482] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:20.020115 3435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:20.020661 3435 raft_consensus.cc:504] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa4cd3592af4eeeb38b2040fd892da9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42271 } }
I20260925 10:12:20.020792 3435 leader_election.cc:302] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daa4cd3592af4eeeb38b2040fd892da9; no voters:
I20260925 10:12:20.021003 3435 leader_election.cc:288] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:20.021028 3440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:20.021137 3440 raft_consensus.cc:686] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9 [term 1 LEADER]: Becoming Leader. State: Replica: daa4cd3592af4eeeb38b2040fd892da9, State: Running, Role: LEADER
I20260925 10:12:20.021267 3435 sys_catalog.cc:564] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:20.021239 3440 consensus_queue.cc:237] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa4cd3592af4eeeb38b2040fd892da9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42271 } }
I20260925 10:12:20.021579 3440 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daa4cd3592af4eeeb38b2040fd892da9. Latest consensus state: current_term: 1 leader_uuid: "daa4cd3592af4eeeb38b2040fd892da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa4cd3592af4eeeb38b2040fd892da9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42271 } } }
I20260925 10:12:20.021646 3440 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:20.021873 3445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:20.022630 3445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:20.023283 3441 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daa4cd3592af4eeeb38b2040fd892da9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa4cd3592af4eeeb38b2040fd892da9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42271 } } }
I20260925 10:12:20.023352 3441 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:20.024405 3445 catalog_manager.cc:1347] Generated new cluster ID: 672162a05d2a48848d5cdafc14fd8c89
I20260925 10:12:20.024456 3445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:20.025039 3439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:20.018446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:60227 (local address 127.0.0.1:42271)
0925 10:12:20.018626 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:20.018633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:20.018658 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:12:20.019485 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:20.019492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:20.019539 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:20.019652 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:20.019948 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:20.021404 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:20.022569 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:20.023164 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:20.023279 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:20.024392 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:20.024396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:20.024396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:20.024584 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:20.024615 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:20.024625 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:20.024732 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:20.024833 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 10:12:20.024863 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260925 10:12:20.025933 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:20.031299 3439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:20.028395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:54201 (local address 127.0.0.1:42271)
0925 10:12:20.028432 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:20.028434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:20.028569 (+ 135us) server_negotiation.cc:407] Connection header received
0925 10:12:20.028642 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:20.028643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:20.028666 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:20.028715 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:20.028870 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:20.029343 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:20.030059 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:20.030286 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:20.030366 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:20.031056 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:20.031058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:20.031058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:20.031063 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:20.031076 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:20.031080 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:20.031137 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:20.031212 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 10:12:20.031223 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:12:20.036808 3445 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:20.037504 3445 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:20.044921 3445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daa4cd3592af4eeeb38b2040fd892da9: Generated new TSK 0
I20260925 10:12:20.045183 3445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:20.047550 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 3367
W20260925 10:12:20.052170 3352 connection.cc:535] client connection to 127.0.0.1:42271 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42271): Connection reset by peer (error 104)
2026-09-25T10:12:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:20 dist-test-slave-s6tm krb5kdc[3466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:20 dist-test-slave-s6tm krb5kdc[3466](info): set up 2 sockets
Sep 25 10:12:20 dist-test-slave-s6tm krb5kdc[3466](info): commencing operation
krb5kdc: starting...
W20260925 10:12:21.113544 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:21 dist-test-slave-s6tm krb5kdc[3466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:12:21.237334 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:21.321410 3479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:21.321619 3479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:21.321641 3479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:21.322834 3479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:21.322880 3479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:21.322894 3479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:21.322907 3479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:21.322919 3479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:21.324571 3479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45551
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:21.324828 3479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:21.325055 3479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:21.329933 3484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:21.329933 3487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:21.329962 3485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:21.330439 3479 server_base.cc:1034] running on GCE node
I20260925 10:12:21.330665 3479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:21.330940 3479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:21.332118 3479 hybrid_clock.cc:648] HybridClock initialized: now 1790331141332097 us; error 41 us; skew 500 ppm
Sep 25 10:12:21 dist-test-slave-s6tm krb5kdc[3466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:21.334542 3479 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:12:21.335007 3479 webserver.cc:460] Webserver started at http://127.0.0.1:36547/ using document root <none> and password file <none>
I20260925 10:12:21.335251 3479 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:21.335317 3479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:21.335461 3479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:21.336582 3479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "7819772c8991440a889ed1fb6370cd81"
format_stamp: "Formatted at 2026-09-25 10:12:21 on dist-test-slave-s6tm"
server_key: "eede0a9067432cd694ba02497926cc36"
server_key_iv: "78a3999419d285991e9a50d95efb4bf7"
server_key_version: "encryptionkey@0"
I20260925 10:12:21.336935 3479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "7819772c8991440a889ed1fb6370cd81"
format_stamp: "Formatted at 2026-09-25 10:12:21 on dist-test-slave-s6tm"
server_key: "eede0a9067432cd694ba02497926cc36"
server_key_iv: "78a3999419d285991e9a50d95efb4bf7"
server_key_version: "encryptionkey@0"
I20260925 10:12:21.338311 3479 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:12:21.339092 3494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:21.339396 3479 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:21.339499 3479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "7819772c8991440a889ed1fb6370cd81"
format_stamp: "Formatted at 2026-09-25 10:12:21 on dist-test-slave-s6tm"
server_key: "eede0a9067432cd694ba02497926cc36"
server_key_iv: "78a3999419d285991e9a50d95efb4bf7"
server_key_version: "encryptionkey@0"
I20260925 10:12:21.339574 3479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:21.343055 3479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:21.345225 3479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:21.345340 3479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:21.349368 3479 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45551
I20260925 10:12:21.349396 3546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45551 every 8 connection(s)
I20260925 10:12:21.349742 3479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:21.350632 3547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:21.352408 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 3479
I20260925 10:12:21.352586 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:21.352788 14501 external_mini_cluster.cc:1428] Setting key c4f420ba4d6906fcbe902863530ce61c
I20260925 10:12:21.353399 3547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81: Bootstrap starting.
I20260925 10:12:21.354153 3547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:21.354488 3547 log.cc:824] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:21.355566 3547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81: No bootstrap required, opened a new log
I20260925 10:12:21.355572 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:21.353332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:50107 (local address 127.0.0.1:45551)
0925 10:12:21.353534 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:21.353544 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:21.353578 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:12:21.354219 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:21.354229 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:21.354268 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:21.354452 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:21.355130 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:21.355134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:21.355134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:21.355137 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:21.355173 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:21.355183 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:21.355292 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:21.355331 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 10:12:21.355374 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260925 10:12:21.356611 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:21.358209 3547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7819772c8991440a889ed1fb6370cd81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45551 } }
I20260925 10:12:21.358402 3547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:21.358493 3547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7819772c8991440a889ed1fb6370cd81, State: Initialized, Role: FOLLOWER
I20260925 10:12:21.358650 3547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7819772c8991440a889ed1fb6370cd81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45551 } }
I20260925 10:12:21.358767 3547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:21.358841 3547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:21.358906 3547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:21.359561 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:21.358415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.146:45597 (local address 127.0.0.1:45551)
0925 10:12:21.358456 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:21.358458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:21.358641 (+ 183us) server_negotiation.cc:407] Connection header received
0925 10:12:21.358709 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:21.358710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:21.358714 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:21.358793 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:21.359345 (+ 552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:21.359346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:21.359346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:21.359350 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:21.359363 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:21.359367 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:21.359411 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:21.359465 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 10:12:21.359476 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:12:21.359762 3547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7819772c8991440a889ed1fb6370cd81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45551 } }
I20260925 10:12:21.359870 3547 leader_election.cc:302] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7819772c8991440a889ed1fb6370cd81; no voters:
I20260925 10:12:21.360069 3547 leader_election.cc:288] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:21.360090 3557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:21.360208 3557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81 [term 1 LEADER]: Becoming Leader. State: Replica: 7819772c8991440a889ed1fb6370cd81, State: Running, Role: LEADER
I20260925 10:12:21.360334 3547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:21.360361 3557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7819772c8991440a889ed1fb6370cd81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45551 } }
I20260925 10:12:21.360692 3559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7819772c8991440a889ed1fb6370cd81. Latest consensus state: current_term: 1 leader_uuid: "7819772c8991440a889ed1fb6370cd81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7819772c8991440a889ed1fb6370cd81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45551 } } }
I20260925 10:12:21.360790 3559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:21.360958 3558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7819772c8991440a889ed1fb6370cd81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7819772c8991440a889ed1fb6370cd81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45551 } } }
I20260925 10:12:21.361048 3558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:21.361006 3563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:21.361786 3563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:21.363523 3563 catalog_manager.cc:1347] Generated new cluster ID: 4704ad3df74546028ca2d0cad0cbfad2
I20260925 10:12:21.363579 3563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:21.387776 3563 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:21.388023 3563 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:21.397238 3563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7819772c8991440a889ed1fb6370cd81: Generated new TSK 0
I20260925 10:12:21.397454 3563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:21.405544 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 3479
2026-09-25T10:12:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:21 dist-test-slave-s6tm krb5kdc[3578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:21 dist-test-slave-s6tm krb5kdc[3578](info): set up 2 sockets
Sep 25 10:12:21 dist-test-slave-s6tm krb5kdc[3578](info): commencing operation
krb5kdc: starting...
W20260925 10:12:22.449524 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:22 dist-test-slave-s6tm krb5kdc[3578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:12:22.575006 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:38739
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:35871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:38739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:22.656808 3591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:22.657018 3591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:22.657042 3591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:22.658160 3591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:22.658200 3591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:22.658214 3591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:22.658226 3591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:22.658239 3591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:22.659679 3591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:35871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:38739
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:38739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:22.659878 3591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:22.660137 3591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:12:22.665127 3591 server_base.cc:1034] running on GCE node
W20260925 10:12:22.665102 3597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:22.665097 3599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:22.665097 3596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:22.665571 3591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:22.665890 3591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:22.667058 3591 hybrid_clock.cc:648] HybridClock initialized: now 1790331142667052 us; error 36 us; skew 500 ppm
Sep 25 10:12:22 dist-test-slave-s6tm krb5kdc[3578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331142, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:22.674841 3591 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:12:22.675328 3591 webserver.cc:460] Webserver started at http://127.14.41.126:45677/ using document root <none> and password file <none>
I20260925 10:12:22.675570 3591 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:22.675645 3591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:22.675814 3591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:22.676993 3591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "a262a716d8dd494589f5149f888247ba"
format_stamp: "Formatted at 2026-09-25 10:12:22 on dist-test-slave-s6tm"
server_key: "5042efde8fcc142e722d9b6ba1f142a7"
server_key_iv: "d42e1df4c2f9f3f736cf6a444e2bf9a5"
server_key_version: "encryptionkey@0"
I20260925 10:12:22.677372 3591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "a262a716d8dd494589f5149f888247ba"
format_stamp: "Formatted at 2026-09-25 10:12:22 on dist-test-slave-s6tm"
server_key: "5042efde8fcc142e722d9b6ba1f142a7"
server_key_iv: "d42e1df4c2f9f3f736cf6a444e2bf9a5"
server_key_version: "encryptionkey@0"
I20260925 10:12:22.678772 3591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:12:22.679515 3606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:22.679685 3591 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:12:22.679834 3591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "a262a716d8dd494589f5149f888247ba"
format_stamp: "Formatted at 2026-09-25 10:12:22 on dist-test-slave-s6tm"
server_key: "5042efde8fcc142e722d9b6ba1f142a7"
server_key_iv: "d42e1df4c2f9f3f736cf6a444e2bf9a5"
server_key_version: "encryptionkey@0"
I20260925 10:12:22.679921 3591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:22.695617 3591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:22.698195 3591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:22.698374 3591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:22.702452 3591 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:38739
I20260925 10:12:22.702494 3658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:38739 every 8 connection(s)
I20260925 10:12:22.702943 3591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:22.703861 3659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:22.706341 3659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba: Bootstrap starting.
I20260925 10:12:22.706871 3659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:22.707109 3659 log.cc:824] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:22.707773 3659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba: No bootstrap required, opened a new log
I20260925 10:12:22.709499 3659 raft_consensus.cc:348] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a262a716d8dd494589f5149f888247ba" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 38739 } }
I20260925 10:12:22.709641 3659 raft_consensus.cc:374] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:22.709671 3659 raft_consensus.cc:729] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a262a716d8dd494589f5149f888247ba, State: Initialized, Role: FOLLOWER
I20260925 10:12:22.709810 3659 consensus_queue.cc:260] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a262a716d8dd494589f5149f888247ba" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 38739 } }
I20260925 10:12:22.709901 3659 raft_consensus.cc:388] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:22.709925 3659 raft_consensus.cc:482] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:22.709954 3659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:22.710237 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 3591
I20260925 10:12:22.710369 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:22.710563 14501 external_mini_cluster.cc:1428] Setting key 7a68c5f4a5e63e045807b1418bdb688d
I20260925 10:12:22.710922 3659 raft_consensus.cc:504] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a262a716d8dd494589f5149f888247ba" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 38739 } }
I20260925 10:12:22.711040 3659 leader_election.cc:302] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a262a716d8dd494589f5149f888247ba; no voters:
I20260925 10:12:22.711216 3659 leader_election.cc:288] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:22.711272 3663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:22.711467 3663 raft_consensus.cc:686] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba [term 1 LEADER]: Becoming Leader. State: Replica: a262a716d8dd494589f5149f888247ba, State: Running, Role: LEADER
I20260925 10:12:22.711527 3659 sys_catalog.cc:564] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:22.711596 3663 consensus_queue.cc:237] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a262a716d8dd494589f5149f888247ba" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 38739 } }
I20260925 10:12:22.712141 3665 sys_catalog.cc:455] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a262a716d8dd494589f5149f888247ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a262a716d8dd494589f5149f888247ba" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 38739 } } }
I20260925 10:12:22.712242 3665 sys_catalog.cc:458] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:22.712423 3666 sys_catalog.cc:455] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader a262a716d8dd494589f5149f888247ba. Latest consensus state: current_term: 1 leader_uuid: "a262a716d8dd494589f5149f888247ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a262a716d8dd494589f5149f888247ba" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 38739 } } }
I20260925 10:12:22.712488 3666 sys_catalog.cc:458] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:22.712603 3673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:22.713109 3673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:22.715633 3673 catalog_manager.cc:1347] Generated new cluster ID: ea150e4da79749e8894718f6030588b0
I20260925 10:12:22.715688 3673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:12:22 dist-test-slave-s6tm krb5kdc[3578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:12:22.722419 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:22.711088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44291 (local address 127.14.41.126:38739)
0925 10:12:22.711329 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:22.711339 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:22.711375 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:12:22.711503 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:22.711512 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:22.711575 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:22.712606 (+ 1031us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:22.712731 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:22.714035 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:22.715119 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:22.715704 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:22.715808 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:22.718378 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:22.718383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:22.718383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:22.718396 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:22.720795 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:22.721193 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:22.721197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:22.721197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:22.721307 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:22.721548 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:22.721549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:22.721549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:22.721815 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:22.722004 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:22.722090 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 10:12:22.722193 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":123,"threads_started":1}
I20260925 10:12:22.723381 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:22.734266 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:22.724803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37655 (local address 127.14.41.126:38739)
0925 10:12:22.724841 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:22.724843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:22.728120 (+ 3277us) server_negotiation.cc:407] Connection header received
0925 10:12:22.728196 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:22.728197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:22.728207 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:22.728253 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:22.728797 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:22.729313 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:22.730032 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:22.730262 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:22.730337 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:22.731551 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:22.731553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:22.731553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:22.731560 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:22.733049 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:22.733431 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:22.733433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:22.733433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:22.733491 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:22.733727 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:22.733728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:22.733728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:22.733840 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:22.733965 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:22.734091 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 10:12:22.734155 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:12:22.738106 3673 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:22.738620 3673 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:22.741175 3673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a262a716d8dd494589f5149f888247ba: Generated new TSK 0
I20260925 10:12:22.741377 3673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:22.746635 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 3591
W20260925 10:12:22.750959 3576 connection.cc:535] client connection to 127.14.41.126:38739 recv error: Network error: failed to read from TLS socket (remote: 127.14.41.126:38739): Connection reset by peer (error 104)
2026-09-25T10:12:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:22 dist-test-slave-s6tm krb5kdc[3690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:22 dist-test-slave-s6tm krb5kdc[3690](info): set up 2 sockets
Sep 25 10:12:22 dist-test-slave-s6tm krb5kdc[3690](info): commencing operation
krb5kdc: starting...
W20260925 10:12:23.785341 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:23 dist-test-slave-s6tm krb5kdc[3690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:12:23.909297 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:34393
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:35337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:34393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:23.992385 3703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:23.992655 3703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:23.992697 3703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:23.993891 3703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:23.993969 3703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:23.993997 3703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:23.994028 3703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:23.994064 3703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:23.995689 3703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:35337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:34393
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:34393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:23.995998 3703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:23.996237 3703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:24.002206 3708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:24.002270 3709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:24.002228 3711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:24.002280 3703 server_base.cc:1034] running on GCE node
I20260925 10:12:24.002866 3703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:24.003225 3703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:24.004423 3703 hybrid_clock.cc:648] HybridClock initialized: now 1790331144004401 us; error 43 us; skew 500 ppm
Sep 25 10:12:24 dist-test-slave-s6tm krb5kdc[3690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331144, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:24.007218 3703 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:12:24.007750 3703 webserver.cc:460] Webserver started at http://127.14.41.126:46409/ using document root <none> and password file <none>
I20260925 10:12:24.008170 3703 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:24.008260 3703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:24.008422 3703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:24.009624 3703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "271dbdeb12f04a47a5b81f876fa2e749"
format_stamp: "Formatted at 2026-09-25 10:12:24 on dist-test-slave-s6tm"
server_key: "92ba9d8560f60492c3d1afb4aca5fb92"
server_key_iv: "3b081211bac23f7bb523c42ef3fa5b2c"
server_key_version: "encryptionkey@0"
I20260925 10:12:24.010035 3703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "271dbdeb12f04a47a5b81f876fa2e749"
format_stamp: "Formatted at 2026-09-25 10:12:24 on dist-test-slave-s6tm"
server_key: "92ba9d8560f60492c3d1afb4aca5fb92"
server_key_iv: "3b081211bac23f7bb523c42ef3fa5b2c"
server_key_version: "encryptionkey@0"
I20260925 10:12:24.011539 3703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:12:24.012563 3718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:24.012770 3703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:24.012885 3703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "271dbdeb12f04a47a5b81f876fa2e749"
format_stamp: "Formatted at 2026-09-25 10:12:24 on dist-test-slave-s6tm"
server_key: "92ba9d8560f60492c3d1afb4aca5fb92"
server_key_iv: "3b081211bac23f7bb523c42ef3fa5b2c"
server_key_version: "encryptionkey@0"
I20260925 10:12:24.012964 3703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:24.021557 3703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:24.023869 3703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:24.024075 3703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:24.027992 3703 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:34393
I20260925 10:12:24.028028 3770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:34393 every 8 connection(s)
I20260925 10:12:24.028385 3703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:24.029237 3771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:24.031579 3771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749: Bootstrap starting.
I20260925 10:12:24.032258 3771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:24.032547 3771 log.cc:824] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:24.033308 3771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749: No bootstrap required, opened a new log
I20260925 10:12:24.034904 3771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "271dbdeb12f04a47a5b81f876fa2e749" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 34393 } }
I20260925 10:12:24.035043 3771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:24.035066 3771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271dbdeb12f04a47a5b81f876fa2e749, State: Initialized, Role: FOLLOWER
I20260925 10:12:24.035246 3771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "271dbdeb12f04a47a5b81f876fa2e749" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 34393 } }
I20260925 10:12:24.035334 3771 raft_consensus.cc:388] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:24.035378 3771 raft_consensus.cc:482] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:24.035430 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 3703
I20260925 10:12:24.035447 3771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:24.035586 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:24.035816 14501 external_mini_cluster.cc:1428] Setting key b890b7af4adc2eb8e9fb859e868fd1b8
I20260925 10:12:24.036695 3771 raft_consensus.cc:504] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "271dbdeb12f04a47a5b81f876fa2e749" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 34393 } }
I20260925 10:12:24.036808 3771 leader_election.cc:302] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 271dbdeb12f04a47a5b81f876fa2e749; no voters:
I20260925 10:12:24.036957 3771 leader_election.cc:288] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:24.036988 3776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:24.037209 3776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749 [term 1 LEADER]: Becoming Leader. State: Replica: 271dbdeb12f04a47a5b81f876fa2e749, State: Running, Role: LEADER
I20260925 10:12:24.037217 3771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:24.037366 3776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "271dbdeb12f04a47a5b81f876fa2e749" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 34393 } }
I20260925 10:12:24.038662 3778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 271dbdeb12f04a47a5b81f876fa2e749. Latest consensus state: current_term: 1 leader_uuid: "271dbdeb12f04a47a5b81f876fa2e749" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "271dbdeb12f04a47a5b81f876fa2e749" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 34393 } } }
I20260925 10:12:24.038894 3778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:24.038797 3777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "271dbdeb12f04a47a5b81f876fa2e749" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "271dbdeb12f04a47a5b81f876fa2e749" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 34393 } } }
I20260925 10:12:24.039006 3777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749 [sys.catalog]: This master's current role is: LEADER
W20260925 10:12:24.039501 3791 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:12:24.039565 3791 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:12:24.039618 3792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:24.039746 3792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:24.041224 3792 catalog_manager.cc:1347] Generated new cluster ID: 1d5f81ef57d14d61be6c93510d651608
I20260925 10:12:24.041277 3792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:24.042569 3775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:24.036509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48681 (local address 127.14.41.126:34393)
0925 10:12:24.036729 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:24.036739 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:24.036778 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:12:24.036896 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:24.036903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:24.036964 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:24.037112 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:24.037472 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:24.038859 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:24.040026 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:24.040688 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:24.040812 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:24.041811 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:24.041815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:24.041816 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:24.042029 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:24.042069 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:24.042087 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:24.042148 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:24.042322 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 10:12:24.042354 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
I20260925 10:12:24.043726 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:24.051982 3775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:24.048241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53521 (local address 127.14.41.126:34393)
0925 10:12:24.048283 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:24.048285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:24.048450 (+ 165us) server_negotiation.cc:407] Connection header received
0925 10:12:24.048489 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:24.048490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:24.048501 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:24.048565 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:24.048719 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:24.049519 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:24.050520 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:24.050855 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:24.050930 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:24.051671 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:24.051672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:24.051672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:24.051678 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:24.051694 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:24.051699 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:24.051763 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:24.051839 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 10:12:24.051852 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:12:24.070309 3792 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:24.071115 3792 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:24.075455 3792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 271dbdeb12f04a47a5b81f876fa2e749: Generated new TSK 0
I20260925 10:12:24.075690 3792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:24.082562 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 3703
W20260925 10:12:24.087091 3688 connection.cc:535] client connection to 127.14.41.126:34393 recv error: Network error: failed to read from TLS socket (remote: 127.14.41.126:34393): Connection reset by peer (error 104)
2026-09-25T10:12:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:24 dist-test-slave-s6tm krb5kdc[3802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:24 dist-test-slave-s6tm krb5kdc[3802](info): set up 2 sockets
Sep 25 10:12:24 dist-test-slave-s6tm krb5kdc[3802](info): commencing operation
krb5kdc: starting...
W20260925 10:12:25.137753 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:25 dist-test-slave-s6tm krb5kdc[3802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:12:25.263346 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:33299
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:33299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:25.347651 3815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:25.347932 3815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:25.348018 3815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:25.349265 3815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:25.349349 3815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:25.349373 3815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:25.349395 3815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:25.349423 3815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:25.351141 3815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:33299
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:33299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:25.351455 3815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:25.351706 3815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:12:25.357115 3815 server_base.cc:1034] running on GCE node
W20260925 10:12:25.357084 3823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:25.357084 3820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:25.357092 3821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:25.357666 3815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:25.358006 3815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:25.359201 3815 hybrid_clock.cc:648] HybridClock initialized: now 1790331145359174 us; error 39 us; skew 500 ppm
Sep 25 10:12:25 dist-test-slave-s6tm krb5kdc[3802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331145, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:25.361876 3815 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:12:25.362381 3815 webserver.cc:460] Webserver started at http://127.14.41.126:35843/ using document root <none> and password file <none>
I20260925 10:12:25.362628 3815 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:25.362704 3815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:25.362855 3815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:25.364040 3815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "e3d44b791fbc4e27946cdbfdb9ec744c"
format_stamp: "Formatted at 2026-09-25 10:12:25 on dist-test-slave-s6tm"
server_key: "08bb92d678a451cbcd11ce35e47f2b9a"
server_key_iv: "23020d96a01e89da1c23864f01243c47"
server_key_version: "encryptionkey@0"
I20260925 10:12:25.364416 3815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "e3d44b791fbc4e27946cdbfdb9ec744c"
format_stamp: "Formatted at 2026-09-25 10:12:25 on dist-test-slave-s6tm"
server_key: "08bb92d678a451cbcd11ce35e47f2b9a"
server_key_iv: "23020d96a01e89da1c23864f01243c47"
server_key_version: "encryptionkey@0"
I20260925 10:12:25.365833 3815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:12:25.366559 3830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:25.366789 3815 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:12:25.366895 3815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "e3d44b791fbc4e27946cdbfdb9ec744c"
format_stamp: "Formatted at 2026-09-25 10:12:25 on dist-test-slave-s6tm"
server_key: "08bb92d678a451cbcd11ce35e47f2b9a"
server_key_iv: "23020d96a01e89da1c23864f01243c47"
server_key_version: "encryptionkey@0"
I20260925 10:12:25.366983 3815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:25.370849 3815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:25.373024 3815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:25.373199 3815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:25.377415 3815 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:33299
I20260925 10:12:25.377480 3882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:33299 every 8 connection(s)
I20260925 10:12:25.377837 3815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:25.378651 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 3815
I20260925 10:12:25.378798 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:25.378916 3883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:25.379019 14501 external_mini_cluster.cc:1428] Setting key 2291b8fc528e7be1e73be41fce5501b0
I20260925 10:12:25.381851 3886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:25.379613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53669 (local address 127.14.41.126:33299)
0925 10:12:25.379840 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:25.379848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:25.379874 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:12:25.380482 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:25.380489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:25.380519 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:25.380678 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:25.381411 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:25.381414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:25.381415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:25.381419 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:25.381471 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:25.381481 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:25.381562 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:25.381631 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 10:12:25.381674 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":106,"threads_started":1}
I20260925 10:12:25.382323 3883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c: Bootstrap starting.
I20260925 10:12:25.382977 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:25.383116 3883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:25.383409 3883 log.cc:824] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:25.384229 3883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c: No bootstrap required, opened a new log
I20260925 10:12:25.385628 3886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:25.384553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40797 (local address 127.14.41.126:33299)
0925 10:12:25.384615 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:25.384617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:25.384734 (+ 117us) server_negotiation.cc:407] Connection header received
0925 10:12:25.384810 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:25.384811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:25.384817 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:25.384861 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:25.385424 (+ 563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:25.385425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:25.385426 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:25.385430 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:25.385444 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:25.385447 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:25.385497 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:25.385550 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 10:12:25.385565 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:12:25.386574 3883 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d44b791fbc4e27946cdbfdb9ec744c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 33299 } }
I20260925 10:12:25.386719 3883 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:25.386744 3883 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3d44b791fbc4e27946cdbfdb9ec744c, State: Initialized, Role: FOLLOWER
I20260925 10:12:25.386885 3883 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d44b791fbc4e27946cdbfdb9ec744c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 33299 } }
I20260925 10:12:25.386968 3883 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:25.387013 3883 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:25.387064 3883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:25.387617 3883 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d44b791fbc4e27946cdbfdb9ec744c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 33299 } }
I20260925 10:12:25.387734 3883 leader_election.cc:302] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3d44b791fbc4e27946cdbfdb9ec744c; no voters:
I20260925 10:12:25.387900 3883 leader_election.cc:288] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:25.387964 3893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:25.388136 3893 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c [term 1 LEADER]: Becoming Leader. State: Replica: e3d44b791fbc4e27946cdbfdb9ec744c, State: Running, Role: LEADER
I20260925 10:12:25.388192 3883 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:25.388319 3893 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d44b791fbc4e27946cdbfdb9ec744c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 33299 } }
I20260925 10:12:25.388680 3894 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3d44b791fbc4e27946cdbfdb9ec744c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d44b791fbc4e27946cdbfdb9ec744c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 33299 } } }
I20260925 10:12:25.388684 3895 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3d44b791fbc4e27946cdbfdb9ec744c. Latest consensus state: current_term: 1 leader_uuid: "e3d44b791fbc4e27946cdbfdb9ec744c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d44b791fbc4e27946cdbfdb9ec744c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 33299 } } }
I20260925 10:12:25.388827 3894 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:25.388898 3895 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:25.389091 3901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:25.389583 3901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:25.391196 3901 catalog_manager.cc:1347] Generated new cluster ID: 5bbff9107141498988f6ed8b0351cf86
I20260925 10:12:25.391253 3901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:25.411077 3901 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:25.411322 3901 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:25.417357 3901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3d44b791fbc4e27946cdbfdb9ec744c: Generated new TSK 0
I20260925 10:12:25.417564 3901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:25.427831 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 3815
2026-09-25T10:12:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:25 dist-test-slave-s6tm krb5kdc[3914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:25 dist-test-slave-s6tm krb5kdc[3914](info): set up 2 sockets
Sep 25 10:12:25 dist-test-slave-s6tm krb5kdc[3914](info): commencing operation
krb5kdc: starting...
W20260925 10:12:26.501499 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:26 dist-test-slave-s6tm krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:12:26.628084 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:42137
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:36501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:42137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:26.710515 3927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:26.710742 3927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:26.710767 3927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:26.712072 3927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:26.712122 3927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:26.712136 3927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:26.712149 3927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:26.712162 3927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:26.713961 3927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:36501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:42137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:42137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:26.714255 3927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:26.714514 3927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:26.719645 3932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:26.719646 3935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:26.719965 3933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:26.719877 3927 server_base.cc:1034] running on GCE node
I20260925 10:12:26.720293 3927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:26.720604 3927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:26.721783 3927 hybrid_clock.cc:648] HybridClock initialized: now 1790331146721770 us; error 41 us; skew 500 ppm
Sep 25 10:12:26 dist-test-slave-s6tm krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331146, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:26.724525 3927 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:12:26.724987 3927 webserver.cc:460] Webserver started at http://127.14.41.126:45215/ using document root <none> and password file <none>
I20260925 10:12:26.725260 3927 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:26.725337 3927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:26.725495 3927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:26.726584 3927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "91728ecde9fe474aa3cb0d26a8de3360"
format_stamp: "Formatted at 2026-09-25 10:12:26 on dist-test-slave-s6tm"
server_key: "a68452ef6fbf1130bff85fd28940d87c"
server_key_iv: "48c0eadfab9f1d73bb470c4fa8ff3023"
server_key_version: "encryptionkey@0"
I20260925 10:12:26.726964 3927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "91728ecde9fe474aa3cb0d26a8de3360"
format_stamp: "Formatted at 2026-09-25 10:12:26 on dist-test-slave-s6tm"
server_key: "a68452ef6fbf1130bff85fd28940d87c"
server_key_iv: "48c0eadfab9f1d73bb470c4fa8ff3023"
server_key_version: "encryptionkey@0"
I20260925 10:12:26.728461 3927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:12:26.729230 3942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:26.729435 3927 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:26.729547 3927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "91728ecde9fe474aa3cb0d26a8de3360"
format_stamp: "Formatted at 2026-09-25 10:12:26 on dist-test-slave-s6tm"
server_key: "a68452ef6fbf1130bff85fd28940d87c"
server_key_iv: "48c0eadfab9f1d73bb470c4fa8ff3023"
server_key_version: "encryptionkey@0"
I20260925 10:12:26.729626 3927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:26.741165 3927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:26.743243 3927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:26.743422 3927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:26.747565 3927 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:42137
I20260925 10:12:26.747601 3994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:42137 every 8 connection(s)
I20260925 10:12:26.748029 3927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:26.748955 3995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:26.751317 3995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360: Bootstrap starting.
I20260925 10:12:26.751969 3995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:26.752300 3995 log.cc:824] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:26.753019 3995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360: No bootstrap required, opened a new log
I20260925 10:12:26.753392 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 3927
I20260925 10:12:26.753530 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:26.753742 14501 external_mini_cluster.cc:1428] Setting key 8cae78c545953b1a95d275f8a36af256
I20260925 10:12:26.755136 3995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91728ecde9fe474aa3cb0d26a8de3360" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42137 } }
I20260925 10:12:26.755317 3995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:26.755359 3995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91728ecde9fe474aa3cb0d26a8de3360, State: Initialized, Role: FOLLOWER
I20260925 10:12:26.755549 3995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91728ecde9fe474aa3cb0d26a8de3360" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42137 } }
I20260925 10:12:26.755643 3995 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:26.755687 3995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:26.755739 3995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:26.756803 3995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91728ecde9fe474aa3cb0d26a8de3360" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42137 } }
I20260925 10:12:26.756934 3995 leader_election.cc:302] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91728ecde9fe474aa3cb0d26a8de3360; no voters:
I20260925 10:12:26.757131 3995 leader_election.cc:288] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:26.757154 4000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:26.757272 4000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360 [term 1 LEADER]: Becoming Leader. State: Replica: 91728ecde9fe474aa3cb0d26a8de3360, State: Running, Role: LEADER
I20260925 10:12:26.757414 3995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:26.757417 4000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91728ecde9fe474aa3cb0d26a8de3360" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42137 } }
I20260925 10:12:26.758090 4001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91728ecde9fe474aa3cb0d26a8de3360. Latest consensus state: current_term: 1 leader_uuid: "91728ecde9fe474aa3cb0d26a8de3360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91728ecde9fe474aa3cb0d26a8de3360" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42137 } } }
I20260925 10:12:26.758239 4001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:26.758446 4002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91728ecde9fe474aa3cb0d26a8de3360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91728ecde9fe474aa3cb0d26a8de3360" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 42137 } } }
I20260925 10:12:26.758577 4002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:26.758776 4009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:26.759371 4009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 10:12:26 dist-test-slave-s6tm krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:12:26.761915 4009 catalog_manager.cc:1347] Generated new cluster ID: 027b256fe0fb499aa27692e7e2ee1501
I20260925 10:12:26.761971 4009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:26.765442 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:26.754313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33869 (local address 127.14.41.126:42137)
0925 10:12:26.754508 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:26.754517 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:26.754544 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:12:26.755116 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:26.755122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:26.755172 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:26.755345 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:26.756228 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:26.757232 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:26.758369 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:26.759084 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:26.759235 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:26.761987 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:26.761991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:26.761992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:26.762005 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:26.763650 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:26.764164 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:26.764167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:26.764167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:26.764295 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:26.764526 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:26.764527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:26.764528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:26.764788 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:26.764974 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:26.765052 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 10:12:26.765200 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":103,"threads_started":1}
I20260925 10:12:26.766440 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:26.774190 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:26.767858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54421 (local address 127.14.41.126:42137)
0925 10:12:26.767899 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:26.767901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:26.768012 (+ 111us) server_negotiation.cc:407] Connection header received
0925 10:12:26.768085 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:26.768086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:26.768095 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:26.768139 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:26.768678 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:26.769158 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:26.769896 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:26.770176 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:26.770277 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:26.771483 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:26.771484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:26.771485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:26.771492 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:26.773013 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:26.773392 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:26.773393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:26.773393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:26.773454 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:26.773657 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:26.773658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:26.773659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:26.773778 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:26.773888 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:26.773999 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 10:12:26.774077 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:12:26.789161 4009 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:26.789765 4009 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:26.795612 4009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91728ecde9fe474aa3cb0d26a8de3360: Generated new TSK 0
I20260925 10:12:26.795854 4009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:26.806450 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 3927
2026-09-25T10:12:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:26 dist-test-slave-s6tm krb5kdc[4026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:26 dist-test-slave-s6tm krb5kdc[4026](info): set up 2 sockets
Sep 25 10:12:26 dist-test-slave-s6tm krb5kdc[4026](info): commencing operation
krb5kdc: starting...
W20260925 10:12:27.857650 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:27 dist-test-slave-s6tm krb5kdc[4026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:12:27.984025 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:35805
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:35805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:28.071667 4039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:28.071990 4039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:28.072073 4039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:28.073513 4039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:28.073581 4039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:28.073597 4039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:28.073611 4039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:28.073623 4039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:28.075309 4039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:35805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:35805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:28.075563 4039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:28.075762 4039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:28.081542 4044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:28.081553 4047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:28.081960 4045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:28.082293 4039 server_base.cc:1034] running on GCE node
I20260925 10:12:28.082592 4039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:28.082911 4039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:28.084079 4039 hybrid_clock.cc:648] HybridClock initialized: now 1790331148084049 us; error 40 us; skew 500 ppm
Sep 25 10:12:28 dist-test-slave-s6tm krb5kdc[4026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331148, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:28.094861 4039 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:12:28.096107 4039 webserver.cc:460] Webserver started at http://127.14.41.126:46135/ using document root <none> and password file <none>
I20260925 10:12:28.096407 4039 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:28.096489 4039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:28.096668 4039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:28.097875 4039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "e3fd07065d164f51807a273e6e109aea"
format_stamp: "Formatted at 2026-09-25 10:12:28 on dist-test-slave-s6tm"
server_key: "b74239672f85b9f0e5f3809df9d449e0"
server_key_iv: "82fbe8caf68ff357e52dc7d5475f50a3"
server_key_version: "encryptionkey@0"
I20260925 10:12:28.098310 4039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "e3fd07065d164f51807a273e6e109aea"
format_stamp: "Formatted at 2026-09-25 10:12:28 on dist-test-slave-s6tm"
server_key: "b74239672f85b9f0e5f3809df9d449e0"
server_key_iv: "82fbe8caf68ff357e52dc7d5475f50a3"
server_key_version: "encryptionkey@0"
I20260925 10:12:28.100404 4039 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260925 10:12:28.101408 4054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:28.101643 4039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:28.101769 4039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "e3fd07065d164f51807a273e6e109aea"
format_stamp: "Formatted at 2026-09-25 10:12:28 on dist-test-slave-s6tm"
server_key: "b74239672f85b9f0e5f3809df9d449e0"
server_key_iv: "82fbe8caf68ff357e52dc7d5475f50a3"
server_key_version: "encryptionkey@0"
I20260925 10:12:28.101861 4039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:28.120740 4039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:28.124732 4039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:28.124931 4039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:28.130339 4039 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:35805
I20260925 10:12:28.130380 4106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:35805 every 8 connection(s)
I20260925 10:12:28.130797 4039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:28.132166 4107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:28.135669 4107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea: Bootstrap starting.
I20260925 10:12:28.136466 4107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:28.136827 4107 log.cc:824] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:28.138207 4107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea: No bootstrap required, opened a new log
I20260925 10:12:28.140413 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 4039
I20260925 10:12:28.140542 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:28.140720 14501 external_mini_cluster.cc:1428] Setting key 9d68134d05af93dacfd9aab7d3fe63ca
I20260925 10:12:28.141091 4107 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3fd07065d164f51807a273e6e109aea" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 35805 } }
I20260925 10:12:28.141295 4107 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:28.141332 4107 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fd07065d164f51807a273e6e109aea, State: Initialized, Role: FOLLOWER
I20260925 10:12:28.141510 4107 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3fd07065d164f51807a273e6e109aea" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 35805 } }
I20260925 10:12:28.141616 4107 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:28.141656 4107 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:28.141726 4107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:28.142758 4107 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3fd07065d164f51807a273e6e109aea" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 35805 } }
I20260925 10:12:28.142928 4107 leader_election.cc:302] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3fd07065d164f51807a273e6e109aea; no voters:
I20260925 10:12:28.143169 4107 leader_election.cc:288] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:28.143201 4112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:28.143319 4112 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea [term 1 LEADER]: Becoming Leader. State: Replica: e3fd07065d164f51807a273e6e109aea, State: Running, Role: LEADER
I20260925 10:12:28.143411 4112 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3fd07065d164f51807a273e6e109aea" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 35805 } }
I20260925 10:12:28.143496 4107 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:28.143800 4112 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3fd07065d164f51807a273e6e109aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3fd07065d164f51807a273e6e109aea" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 35805 } } }
I20260925 10:12:28.143877 4112 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:28.144043 4112 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3fd07065d164f51807a273e6e109aea. Latest consensus state: current_term: 1 leader_uuid: "e3fd07065d164f51807a273e6e109aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3fd07065d164f51807a273e6e109aea" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 35805 } } }
I20260925 10:12:28.144119 4117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:28.144197 4112 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:28.144897 4117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:28.147019 4117 catalog_manager.cc:1347] Generated new cluster ID: bc1147dea17842639907787df0817a30
I20260925 10:12:28.147097 4117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:28.152218 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:28.141387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55545 (local address 127.14.41.126:35805)
0925 10:12:28.141658 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:28.141671 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:28.145296 (+ 3625us) server_negotiation.cc:407] Connection header received
0925 10:12:28.145464 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:28.145472 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:28.145543 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:28.145721 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:28.145944 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:28.147365 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:28.148691 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:28.149373 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:28.149487 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:28.150778 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:28.150785 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:28.150785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:28.151044 (+ 259us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:28.151091 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:28.151109 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:28.151175 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:28.151500 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 10:12:28.151587 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":115,"threads_started":1}
I20260925 10:12:28.153787 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:28.160287 4117 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:28.161067 4117 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:28.163671 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:28.156313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52869 (local address 127.14.41.126:35805)
0925 10:12:28.156368 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:28.156370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:28.159678 (+ 3308us) server_negotiation.cc:407] Connection header received
0925 10:12:28.159719 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:28.159721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:28.159735 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:28.159822 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:28.160286 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:28.161008 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:28.162130 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:28.162465 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:28.162538 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:28.163364 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:28.163365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:28.163366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:28.163374 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:28.163391 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:28.163396 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:28.163452 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:28.163534 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 10:12:28.163584 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 10:12:28.168581 4117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3fd07065d164f51807a273e6e109aea: Generated new TSK 0
I20260925 10:12:28.168934 4117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:28.174347 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 4039
2026-09-25T10:12:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:28 dist-test-slave-s6tm krb5kdc[4138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:28 dist-test-slave-s6tm krb5kdc[4138](info): set up 2 sockets
Sep 25 10:12:28 dist-test-slave-s6tm krb5kdc[4138](info): commencing operation
krb5kdc: starting...
W20260925 10:12:29.289608 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:29 dist-test-slave-s6tm krb5kdc[4138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:12:29.419377 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:38237
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:38011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:38237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:29.504787 4151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:29.505010 4151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:29.505035 4151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:29.506310 4151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:29.506361 4151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:29.506376 4151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:29.506388 4151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:29.506400 4151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:29.508061 4151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:38011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:38237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:38237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:29.508335 4151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:29.508585 4151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:29.513722 4159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:29.513824 4157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:29.513849 4151 server_base.cc:1034] running on GCE node
W20260925 10:12:29.513825 4156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:29.514307 4151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:29.514624 4151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:29.515797 4151 hybrid_clock.cc:648] HybridClock initialized: now 1790331149515772 us; error 60 us; skew 500 ppm
Sep 25 10:12:29 dist-test-slave-s6tm krb5kdc[4138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331149, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:29.518385 4151 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:12:29.518838 4151 webserver.cc:460] Webserver started at http://127.14.41.126:45917/ using document root <none> and password file <none>
I20260925 10:12:29.519037 4151 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:29.519079 4151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:29.519166 4151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:29.520285 4151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "548178c26a5f4aa591d9605eb3683e35"
format_stamp: "Formatted at 2026-09-25 10:12:29 on dist-test-slave-s6tm"
server_key: "0bb7740ea8d15fba62312672bd332d43"
server_key_iv: "161016b0fca1bea6cad9396e0737b1cb"
server_key_version: "encryptionkey@0"
I20260925 10:12:29.520604 4151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "548178c26a5f4aa591d9605eb3683e35"
format_stamp: "Formatted at 2026-09-25 10:12:29 on dist-test-slave-s6tm"
server_key: "0bb7740ea8d15fba62312672bd332d43"
server_key_iv: "161016b0fca1bea6cad9396e0737b1cb"
server_key_version: "encryptionkey@0"
I20260925 10:12:29.521927 4151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:12:29.522794 4166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:29.523025 4151 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:29.523118 4151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "548178c26a5f4aa591d9605eb3683e35"
format_stamp: "Formatted at 2026-09-25 10:12:29 on dist-test-slave-s6tm"
server_key: "0bb7740ea8d15fba62312672bd332d43"
server_key_iv: "161016b0fca1bea6cad9396e0737b1cb"
server_key_version: "encryptionkey@0"
I20260925 10:12:29.523177 4151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:29.526718 4151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:29.529112 4151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:29.529223 4151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:29.533507 4151 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:38237
I20260925 10:12:29.533591 4218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:38237 every 8 connection(s)
I20260925 10:12:29.533944 4151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:29.534479 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 4151
I20260925 10:12:29.534561 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:29.534794 14501 external_mini_cluster.cc:1428] Setting key 219d5e2482fb7590481b0c5897190769
I20260925 10:12:29.535223 4219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:29.537842 4222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:29.535341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50829 (local address 127.14.41.126:38237)
0925 10:12:29.535615 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:29.535624 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:29.535655 (+ 31us) server_negotiation.cc:407] Connection header received
0925 10:12:29.536475 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:29.536485 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:29.536522 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:29.536684 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:29.537408 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:29.537413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:29.537413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:29.537418 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:29.537466 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:29.537480 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:29.537559 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:29.537616 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 10:12:29.537657 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":70,"threads_started":1}
I20260925 10:12:29.539014 4219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35: Bootstrap starting.
I20260925 10:12:29.539101 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:29.539644 4219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:29.540015 4219 log.cc:824] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:29.540887 4219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35: No bootstrap required, opened a new log
I20260925 10:12:29.541648 4222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:29.540565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56293 (local address 127.14.41.126:38237)
0925 10:12:29.540606 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:29.540608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:29.540681 (+ 73us) server_negotiation.cc:407] Connection header received
0925 10:12:29.540759 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:29.540759 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:29.540764 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:29.540808 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:29.541402 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:29.541403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:29.541404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:29.541407 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:29.541421 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:29.541425 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:29.541471 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:29.541536 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 10:12:29.541547 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:12:29.543056 4219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548178c26a5f4aa591d9605eb3683e35" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 38237 } }
I20260925 10:12:29.543233 4219 raft_consensus.cc:374] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:29.543256 4219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 548178c26a5f4aa591d9605eb3683e35, State: Initialized, Role: FOLLOWER
I20260925 10:12:29.543399 4219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548178c26a5f4aa591d9605eb3683e35" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 38237 } }
I20260925 10:12:29.543488 4219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:29.543511 4219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:29.543582 4219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:29.544220 4219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548178c26a5f4aa591d9605eb3683e35" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 38237 } }
I20260925 10:12:29.544335 4219 leader_election.cc:302] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 548178c26a5f4aa591d9605eb3683e35; no voters:
I20260925 10:12:29.544504 4219 leader_election.cc:288] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:29.544566 4229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:29.544763 4229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35 [term 1 LEADER]: Becoming Leader. State: Replica: 548178c26a5f4aa591d9605eb3683e35, State: Running, Role: LEADER
I20260925 10:12:29.544775 4219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:29.544864 4229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548178c26a5f4aa591d9605eb3683e35" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 38237 } }
I20260925 10:12:29.545320 4230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "548178c26a5f4aa591d9605eb3683e35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548178c26a5f4aa591d9605eb3683e35" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 38237 } } }
I20260925 10:12:29.545322 4231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 548178c26a5f4aa591d9605eb3683e35. Latest consensus state: current_term: 1 leader_uuid: "548178c26a5f4aa591d9605eb3683e35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "548178c26a5f4aa591d9605eb3683e35" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 38237 } } }
I20260925 10:12:29.545473 4230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:29.545550 4231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:29.546046 4238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:29.546535 4238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:29.548202 4238 catalog_manager.cc:1347] Generated new cluster ID: b34cb63673d649b69521cfb59291c74f
I20260925 10:12:29.548280 4238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:29.556036 4238 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:29.556274 4238 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:29.563120 4238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 548178c26a5f4aa591d9605eb3683e35: Generated new TSK 0
I20260925 10:12:29.563378 4238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:29.566319 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 4151
2026-09-25T10:12:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1392 ms)
[----------] 18 tests from AuthTokenIssuingTest (24404 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:29 dist-test-slave-s6tm krb5kdc[4250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:29 dist-test-slave-s6tm krb5kdc[4250](info): set up 2 sockets
Sep 25 10:12:29 dist-test-slave-s6tm krb5kdc[4250](info): commencing operation
krb5kdc: starting...
W20260925 10:12:30.621709 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:30 dist-test-slave-s6tm krb5kdc[4250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:12:30.746838 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:36419
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:43427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:36419,127.14.41.125:45017,127.14.41.124:37955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:30.830380 4263 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:12:30.830612 4263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:30.830641 4263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:30.830663 4263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:30.831774 4263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:30.831813 4263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:30.831828 4263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:30.831840 4263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:30.831852 4263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:30.833469 4263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:43427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:36419,127.14.41.125:45017,127.14.41.124:37955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:36419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:30.833690 4263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:30.833881 4263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:30.839058 4269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:30.839077 4271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:30.839170 4263 server_base.cc:1034] running on GCE node
W20260925 10:12:30.839077 4268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:30.839586 4263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:30.839887 4263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:30.841078 4263 hybrid_clock.cc:648] HybridClock initialized: now 1790331150841058 us; error 43 us; skew 500 ppm
Sep 25 10:12:30 dist-test-slave-s6tm krb5kdc[4250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331150, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:30.843688 4263 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:12:30.844197 4263 webserver.cc:460] Webserver started at http://127.14.41.126:43715/ using document root <none> and password file <none>
I20260925 10:12:30.844406 4263 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:30.844448 4263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:30.844544 4263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:30.845587 4263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "dfdb76636ed644359ebc0b4c24a12c8c"
format_stamp: "Formatted at 2026-09-25 10:12:30 on dist-test-slave-s6tm"
server_key: "28130441a20f4011d1444d2bfe154fbc"
server_key_iv: "5f9d412504128c72dd20a257f1260579"
server_key_version: "encryptionkey@0"
I20260925 10:12:30.845912 4263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "dfdb76636ed644359ebc0b4c24a12c8c"
format_stamp: "Formatted at 2026-09-25 10:12:30 on dist-test-slave-s6tm"
server_key: "28130441a20f4011d1444d2bfe154fbc"
server_key_iv: "5f9d412504128c72dd20a257f1260579"
server_key_version: "encryptionkey@0"
I20260925 10:12:30.847211 4263 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:12:30.847992 4278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:30.848189 4263 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:30.848284 4263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "dfdb76636ed644359ebc0b4c24a12c8c"
format_stamp: "Formatted at 2026-09-25 10:12:30 on dist-test-slave-s6tm"
server_key: "28130441a20f4011d1444d2bfe154fbc"
server_key_iv: "5f9d412504128c72dd20a257f1260579"
server_key_version: "encryptionkey@0"
I20260925 10:12:30.848343 4263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:30.863649 4263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:30.866444 4263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:30.866580 4263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:30.870831 4263 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:36419
I20260925 10:12:30.870939 4330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:36419 every 8 connection(s)
I20260925 10:12:30.871284 4263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:30.872219 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 4263
I20260925 10:12:30.872305 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:30.872517 14501 external_mini_cluster.cc:1428] Setting key 02392e6b88256a3bfb6e6701d43f6596
I20260925 10:12:30.872526 4331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:30.874429 4331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.41.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.125@KRBTEST.COM" created.
Sep 25 10:12:30 dist-test-slave-s6tm krb5kdc[4250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331150, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:12:30.887709 4335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:30.875429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45506 (local address 127.14.41.126:36419)
0925 10:12:30.875647 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:30.875657 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:30.875808 (+ 151us) server_negotiation.cc:407] Connection header received
0925 10:12:30.876559 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:30.876563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:30.876585 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:30.876691 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:30.877275 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:30.878178 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:30.879240 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:30.879576 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:30.879714 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:30.883911 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:30.883920 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:30.883920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:30.883962 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:30.885994 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:30.886442 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:30.886445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:30.886445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:30.886527 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:30.886769 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:30.886770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:30.886771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:30.887017 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:30.887116 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:30.887306 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 10:12:30.887454 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":104,"threads_started":1}
I20260925 10:12:30.887715 4334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:30.875397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:36419 (local address 127.0.0.1:45506)
0925 10:12:30.875631 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:30.875676 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:30.876360 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:30.876697 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:30.876699 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:30.876993 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:30.877154 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:30.877159 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:30.878297 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:30.878298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:30.879113 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:30.879116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:30.879699 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:30.879700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:30.879765 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:30.880412 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:30.880421 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:30.883809 (+ 3388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:30.886097 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:30.886103 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:30.886105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:30.886309 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:30.886600 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:30.886601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:30.886601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:30.886693 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:30.887092 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:30.887097 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:30.887161 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 10:12:30.887273 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 10:12:30.887529 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":96,"thread_start_us":118,"threads_started":1}
I20260925 10:12:30.888501 4331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:12:30.888881 4334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:30.888749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:45017 (local address 127.0.0.1:59404)
0925 10:12:30.888807 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:30.888855 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.125:45017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 10:12:30.889024 4281 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.41.125:45017: connect: Connection refused (error 111)
I20260925 10:12:30.889441 4334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:30.889333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:45017 (local address 127.0.0.1:59416)
0925 10:12:30.889384 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:30.889408 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.125:45017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 10:12:30.889552 4331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.125:45017: Network error: Client connection negotiation failed: client connection to 127.14.41.125:45017: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.125.keytab.
Entry for principal kudu/127.14.41.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.125.keytab.
I20260925 10:12:30.899407 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.125:45017
--webserver_interface=127.14.41.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.125.keytab
--principal=kudu/127.14.41.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:43427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:36419,127.14.41.125:45017,127.14.41.124:37955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:12:30.938843 4331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } attempt: 1
I20260925 10:12:30.939707 4334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:30.939419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:45017 (local address 127.0.0.1:59426)
0925 10:12:30.939571 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:30.939638 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.125:45017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260925 10:12:30.940299 4334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:30.940178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:45017 (local address 127.0.0.1:59438)
0925 10:12:30.940228 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:30.940268 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.125:45017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 10:12:30.940426 4331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.125:45017: Network error: Client connection negotiation failed: client connection to 127.14.41.125:45017: connect: Connection refused (error 111)
W20260925 10:12:30.984097 4338 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:12:30.984332 4338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:30.984350 4338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:30.984373 4338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:30.985582 4338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:30.985621 4338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:30.985635 4338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:30.985648 4338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:30.985661 4338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:30.987165 4338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:43427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:36419,127.14.41.125:45017,127.14.41.124:37955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.125.keytab
--principal=kudu/127.14.41.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.125:45017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.41.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:30.987378 4338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:30.987556 4338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:30.992923 4343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:30.992908 4344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:30.992903 4338 server_base.cc:1034] running on GCE node
W20260925 10:12:30.992924 4346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:30.993542 4338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:30.993858 4338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:30.995036 4338 hybrid_clock.cc:648] HybridClock initialized: now 1790331150995012 us; error 42 us; skew 500 ppm
Sep 25 10:12:30 dist-test-slave-s6tm krb5kdc[4250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331150, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:30.997823 4338 init.cc:377] Logged in from keytab as kudu/127.14.41.125@KRBTEST.COM (short username kudu)
I20260925 10:12:30.998334 4338 webserver.cc:460] Webserver started at http://127.14.41.125:41323/ using document root <none> and password file <none>
I20260925 10:12:30.998536 4338 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:30.998584 4338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:30.998679 4338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:30.999778 4338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-1/data/instance:
uuid: "1cff848fecd746399047df658bb7996e"
format_stamp: "Formatted at 2026-09-25 10:12:30 on dist-test-slave-s6tm"
server_key: "35678a8437041fdead5082ab42ed2b32"
server_key_iv: "b5699a3dea59970ee304a2305ff6cf62"
server_key_version: "encryptionkey@0"
I20260925 10:12:31.000164 4338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-1/wal/instance:
uuid: "1cff848fecd746399047df658bb7996e"
format_stamp: "Formatted at 2026-09-25 10:12:30 on dist-test-slave-s6tm"
server_key: "35678a8437041fdead5082ab42ed2b32"
server_key_iv: "b5699a3dea59970ee304a2305ff6cf62"
server_key_version: "encryptionkey@0"
I20260925 10:12:31.001590 4338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:12:31.002482 4353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:31.002704 4338 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:12:31.002794 4338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-1/wal
uuid: "1cff848fecd746399047df658bb7996e"
format_stamp: "Formatted at 2026-09-25 10:12:30 on dist-test-slave-s6tm"
server_key: "35678a8437041fdead5082ab42ed2b32"
server_key_iv: "b5699a3dea59970ee304a2305ff6cf62"
server_key_version: "encryptionkey@0"
I20260925 10:12:31.002862 4338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:31.008569 4331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } attempt: 2
I20260925 10:12:31.009006 4334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.008875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:45017 (local address 127.0.0.1:59440)
0925 10:12:31.008936 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:31.008976 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.125:45017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 10:12:31.009431 4334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.009335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:45017 (local address 127.0.0.1:59442)
0925 10:12:31.009381 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:31.009407 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.125:45017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 10:12:31.009523 4331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.125:45017: Network error: Client connection negotiation failed: client connection to 127.14.41.125:45017: connect: Connection refused (error 111)
I20260925 10:12:31.027113 4338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:31.029844 4338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:31.030049 4338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:31.034217 4338 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.125:45017
I20260925 10:12:31.034250 4405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.125:45017 every 8 connection(s)
I20260925 10:12:31.034652 4338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-1/data/info.pb
I20260925 10:12:31.034888 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 4338
I20260925 10:12:31.035001 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-1/wal/instance
I20260925 10:12:31.035260 14501 external_mini_cluster.cc:1428] Setting key 1f4da0ae1d2e35f4877aa88168c70118
I20260925 10:12:31.036048 4406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:31.037940 4406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.41.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.124@KRBTEST.COM" created.
Sep 25 10:12:31 dist-test-slave-s6tm krb5kdc[4250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331150, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.125@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:12:31.050172 4335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.038761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.14.41.126:36419)
0925 10:12:31.038832 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.038834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.039045 (+ 211us) server_negotiation.cc:407] Connection header received
0925 10:12:31.039659 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.039660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.039672 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.039719 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.040434 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.040951 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.041963 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.042187 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.042279 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.046767 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:31.046769 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:31.046769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:31.046779 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:31.048455 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.049006 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.049007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.049007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.049066 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.049411 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.049412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.049412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.049527 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:31.049584 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:31.049965 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 10:12:31.050056 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:12:31.050432 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.038743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:36419 (local address 127.0.0.1:45514)
0925 10:12:31.038918 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:31.038955 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:31.039536 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:31.039791 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:31.039794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:31.040127 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:31.040332 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:31.040337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.041072 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.041073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:31.041869 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:31.041871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.042301 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.042302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:31.042433 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.043032 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:31.043041 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:31.046649 (+ 3608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:31.048555 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:31.048564 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:31.048566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:31.048924 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:31.049181 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:31.049183 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:31.049183 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:31.049345 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:31.049600 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:31.049607 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:31.049803 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 10:12:31.049930 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 10:12:31.050151 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
I20260925 10:12:31.051119 4406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:12:31 dist-test-slave-s6tm krb5kdc[4250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331150, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.125@KRBTEST.COM for kudu/127.14.41.125@KRBTEST.COM
I20260925 10:12:31.060390 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.051411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:45017 (local address 127.0.0.1:59452)
0925 10:12:31.051644 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:31.051656 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:31.051715 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:31.051888 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:31.051888 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:31.052128 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:31.052196 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:31.052199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.053027 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.053028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:31.053884 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:31.053886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.054418 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.054419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:31.054506 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.054900 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:31.054905 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:31.057266 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:31.059179 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:31.059180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:31.059180 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:31.059347 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:31.059603 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:31.059604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:31.059604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:31.059659 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:31.060037 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:31.060037 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:31.060080 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 10:12:31.060156 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 10:12:31.060261 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215}
I20260925 10:12:31.060410 4411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.051483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.14.41.125:45017)
0925 10:12:31.051640 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.051644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.051708 (+ 64us) server_negotiation.cc:407] Connection header received
0925 10:12:31.051808 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.051810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.051821 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.051880 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.052271 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.052936 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.053990 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.054335 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.054392 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.057375 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:31.057382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:31.057383 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:31.057389 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:31.059028 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.059457 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.059461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.059461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.059523 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.059732 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.059733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.059733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.059887 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:31.060091 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:31.060186 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 10:12:31.060312 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260925 10:12:31.061094 4406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:12:31.061473 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.061354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:37955 (local address 127.0.0.1:54570)
0925 10:12:31.061403 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:31.061449 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:37955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 10:12:31.061619 4354 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.41.124:37955: connect: Connection refused (error 111)
I20260925 10:12:31.062088 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.061971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:37955 (local address 127.0.0.1:54576)
0925 10:12:31.062012 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:31.062070 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:37955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 10:12:31.062189 4406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.124:37955: Network error: Client connection negotiation failed: client connection to 127.14.41.124:37955: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.124.keytab.
Entry for principal kudu/127.14.41.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.124.keytab.
I20260925 10:12:31.062959 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.124:37955
--webserver_interface=127.14.41.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.124.keytab
--principal=kudu/127.14.41.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:43427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:36419,127.14.41.125:45017,127.14.41.124:37955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:12:31.100716 4331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } attempt: 3
Sep 25 10:12:31 dist-test-slave-s6tm krb5kdc[4250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331150, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for kudu/127.14.41.125@KRBTEST.COM
I20260925 10:12:31.108690 4411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.101392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59458 (local address 127.14.41.125:45017)
0925 10:12:31.101465 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.101467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.101479 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:12:31.101551 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.101552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.101570 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.101638 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.101992 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.102503 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.103234 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.103461 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.103561 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.105867 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:31.105869 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:31.105869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:31.105883 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:31.107455 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.107839 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.107840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.107841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.107901 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.108147 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.108148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.108149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.108267 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:31.108347 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:31.108510 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 10:12:31.108588 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:12:31.108690 4334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.101315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:45017 (local address 127.0.0.1:59458)
0925 10:12:31.101370 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:31.101383 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:31.101472 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:31.101645 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:31.101646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:31.101826 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:31.101896 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:31.101898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.102606 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.102608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:31.103144 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:31.103146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.103557 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.103558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:31.103647 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.104229 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:31.104234 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:31.105759 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:31.107568 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:31.107569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:31.107569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:31.107750 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:31.108012 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:31.108014 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:31.108014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:31.108076 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:31.108337 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:31.108338 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:31.108388 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 10:12:31.108478 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 10:12:31.108588 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 10:12:31.109215 4331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:12:31.109643 4334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.109520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:37955 (local address 127.0.0.1:54584)
0925 10:12:31.109583 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:31.109618 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:37955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 10:12:31.110038 4334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.109955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:37955 (local address 127.0.0.1:54596)
0925 10:12:31.109993 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:31.110016 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:37955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 10:12:31.110143 4331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.124:37955: Network error: Client connection negotiation failed: client connection to 127.14.41.124:37955: connect: Connection refused (error 111)
I20260925 10:12:31.111418 4406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } attempt: 1
I20260925 10:12:31.111896 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.111764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:37955 (local address 127.0.0.1:54600)
0925 10:12:31.111834 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:31.111864 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:37955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 10:12:31.112262 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.112169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:37955 (local address 127.0.0.1:54612)
0925 10:12:31.112215 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:31.112240 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:37955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 10:12:31.112361 4406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.124:37955: Network error: Client connection negotiation failed: client connection to 127.14.41.124:37955: connect: Connection refused (error 111)
I20260925 10:12:31.141299 4331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } attempt: 1
I20260925 10:12:31.141832 4334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.141670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:37955 (local address 127.0.0.1:54624)
0925 10:12:31.141747 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:31.141801 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:37955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 10:12:31.142244 4334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.142108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:37955 (local address 127.0.0.1:54636)
0925 10:12:31.142196 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:31.142223 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:37955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260925 10:12:31.142365 4331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.124:37955: Network error: Client connection negotiation failed: client connection to 127.14.41.124:37955: connect: Connection refused (error 111)
W20260925 10:12:31.149351 4413 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:12:31.149616 4413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:31.149658 4413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:31.149703 4413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:31.150909 4413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:31.150990 4413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:31.151017 4413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:31.151044 4413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:31.151072 4413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:31.152880 4413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:43427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:36419,127.14.41.125:45017,127.14.41.124:37955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.124.keytab
--principal=kudu/127.14.41.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.124:37955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.41.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:31.153183 4413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:31.153429 4413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:31.158918 4418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:31.158905 4421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:31.158936 4413 server_base.cc:1034] running on GCE node
W20260925 10:12:31.158905 4419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:31.159411 4413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:31.159760 4413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:31.160918 4413 hybrid_clock.cc:648] HybridClock initialized: now 1790331151160897 us; error 32 us; skew 500 ppm
Sep 25 10:12:31 dist-test-slave-s6tm krb5kdc[4250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331151, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:31.163642 4413 init.cc:377] Logged in from keytab as kudu/127.14.41.124@KRBTEST.COM (short username kudu)
I20260925 10:12:31.164207 4413 webserver.cc:460] Webserver started at http://127.14.41.124:37485/ using document root <none> and password file <none>
I20260925 10:12:31.164451 4413 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:31.164528 4413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:31.164695 4413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:31.165812 4413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-2/data/instance:
uuid: "dea7373b963145f6b7749b677830143f"
format_stamp: "Formatted at 2026-09-25 10:12:31 on dist-test-slave-s6tm"
server_key: "d4aef23c20c581ea78095a6c844b3964"
server_key_iv: "86ce18763925f4fda8064f02fdfa1694"
server_key_version: "encryptionkey@0"
I20260925 10:12:31.166200 4413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-2/wal/instance:
uuid: "dea7373b963145f6b7749b677830143f"
format_stamp: "Formatted at 2026-09-25 10:12:31 on dist-test-slave-s6tm"
server_key: "d4aef23c20c581ea78095a6c844b3964"
server_key_iv: "86ce18763925f4fda8064f02fdfa1694"
server_key_version: "encryptionkey@0"
I20260925 10:12:31.167603 4413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:12:31.168463 4428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:31.168675 4413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:12:31.168792 4413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-2/wal
uuid: "dea7373b963145f6b7749b677830143f"
format_stamp: "Formatted at 2026-09-25 10:12:31 on dist-test-slave-s6tm"
server_key: "d4aef23c20c581ea78095a6c844b3964"
server_key_iv: "86ce18763925f4fda8064f02fdfa1694"
server_key_version: "encryptionkey@0"
I20260925 10:12:31.168871 4413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:31.180512 4406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } attempt: 2
I20260925 10:12:31.181003 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.180864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:37955 (local address 127.0.0.1:54644)
0925 10:12:31.180931 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:31.180974 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:37955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 10:12:31.181461 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.181359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:37955 (local address 127.0.0.1:54656)
0925 10:12:31.181404 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:31.181430 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:37955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 10:12:31.181545 4406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.124:37955: Network error: Client connection negotiation failed: client connection to 127.14.41.124:37955: connect: Connection refused (error 111)
I20260925 10:12:31.197829 4413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:31.200935 4413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:31.201195 4413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:31.205345 4413 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.124:37955
I20260925 10:12:31.205382 4480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.124:37955 every 8 connection(s)
I20260925 10:12:31.205814 4413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-2/data/info.pb
I20260925 10:12:31.207001 4481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:31.209041 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 4413
I20260925 10:12:31.209157 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790330877480195-14501-0/minicluster-data/master-2/wal/instance
I20260925 10:12:31.209365 14501 external_mini_cluster.cc:1428] Setting key fe84d8160aefabc052237046ae61134e
I20260925 10:12:31.209172 4481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:12:31 dist-test-slave-s6tm krb5kdc[4250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331151, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.124@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
Sep 25 10:12:31 dist-test-slave-s6tm krb5kdc[4250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.124@KRBTEST.COM
I20260925 10:12:31.217509 4331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } attempt: 2
I20260925 10:12:31.220269 4335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.210121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45524 (local address 127.14.41.126:36419)
0925 10:12:31.210168 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.210172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.210448 (+ 276us) server_negotiation.cc:407] Connection header received
0925 10:12:31.211144 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.211146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.211160 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.211230 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.211757 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.212347 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.213489 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.213791 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.213876 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.216664 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:31.216665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:31.216666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:31.216675 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:31.218665 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.219174 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.219175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.219176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.219241 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.219547 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.219548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.219548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.219702 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:31.219767 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:31.220095 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 10:12:31.220175 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:12:31.220595 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.210129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:36419 (local address 127.0.0.1:45524)
0925 10:12:31.210323 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:31.210359 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:31.210922 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:31.211245 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:31.211247 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:31.211505 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:31.211667 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:31.211670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.212427 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.212428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:31.213397 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:31.213399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.213895 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.213896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:31.214044 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.214835 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:31.214843 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:31.216559 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:31.218751 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:31.218759 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:31.218761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:31.219093 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:31.219318 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:31.219319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:31.219319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:31.219467 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:31.219759 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:31.219766 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:31.219954 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 10:12:31.220062 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:12:31.220229 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":48,"thread_start_us":104,"threads_started":1}
I20260925 10:12:31.221205 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.210355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.14.41.124:37955)
0925 10:12:31.210508 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.210513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.210555 (+ 42us) server_negotiation.cc:407] Connection header received
0925 10:12:31.210956 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.210970 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.210998 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.211119 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.211829 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.212985 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.213973 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.214394 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.214449 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.217216 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:31.217225 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:31.217225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:31.217232 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:31.219325 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.219729 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.219732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.219732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.219792 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.219999 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.219999 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.220000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.220178 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:31.221028 (+ 850us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:31.221068 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 10:12:31.221134 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260925 10:12:31.221701 4481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:12:31 dist-test-slave-s6tm krb5kdc[4250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331150, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for kudu/127.14.41.124@KRBTEST.COM
Sep 25 10:12:31 dist-test-slave-s6tm krb5kdc[4250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331151, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.124@KRBTEST.COM for kudu/127.14.41.125@KRBTEST.COM
I20260925 10:12:31.226444 4334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.217806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:37955 (local address 127.0.0.1:54674)
0925 10:12:31.217894 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:31.217907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:31.217951 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:31.218185 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:31.218186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:31.218354 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:31.218404 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:31.218407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.219234 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.219236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:31.220065 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:31.220068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.220534 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.220536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:31.220599 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.220989 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:31.220995 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:31.222546 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:31.225299 (+ 2753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:31.225300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:31.225301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:31.225502 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:31.225769 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:31.225771 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:31.225771 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:31.225839 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:31.226140 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:31.226141 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:31.226184 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 10:12:31.226230 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 10:12:31.226329 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 10:12:31.226913 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.217840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54674 (local address 127.14.41.124:37955)
0925 10:12:31.218055 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.218057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.218062 (+ 5us) server_negotiation.cc:407] Connection header received
0925 10:12:31.218078 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.218079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.218087 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.218164 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.218473 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.219137 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.220150 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.220448 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.220742 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.222683 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:31.222684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:31.222685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:31.222692 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:31.225216 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.225590 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.225591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.225592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.225668 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.225917 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.225918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.225918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.226055 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:31.226658 (+ 603us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:31.226703 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 10:12:31.226780 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":120,"threads_started":1}
Sep 25 10:12:31 dist-test-slave-s6tm krb5kdc[4250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:12:31.228659 4331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c: Bootstrap starting.
I20260925 10:12:31.229314 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.222032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:45017 (local address 127.0.0.1:59460)
0925 10:12:31.222126 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:31.222135 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:31.222171 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:31.222322 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:31.222323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:31.222460 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:31.222525 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:31.222527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.223462 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.223464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:31.224039 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:31.224041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.224431 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.224432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:31.224504 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.225130 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:31.225134 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:31.226329 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:31.228249 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:31.228251 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:31.228251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:31.228418 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:31.228635 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:31.228636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:31.228637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:31.228693 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:31.228992 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:31.228993 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:31.229036 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 10:12:31.229129 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 10:12:31.229224 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 10:12:31.229314 4411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.222139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59460 (local address 127.14.41.125:45017)
0925 10:12:31.222184 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.222194 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.222205 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:12:31.222240 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.222240 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.222250 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.222303 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.222616 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.223372 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.224139 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.224351 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.224423 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.226422 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:31.226423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:31.226423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:31.226430 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:31.228171 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.228505 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.228506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.228507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.228569 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.228798 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.228799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.228799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.228916 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:31.228987 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:31.229156 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 10:12:31.229227 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:12:31.229329 4331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:31.229640 4481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:12:31.229692 4331 log.cc:824] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:31.230873 4331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c: No bootstrap required, opened a new log
I20260925 10:12:31.233155 4331 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } } peers { permanent_uuid: "1cff848fecd746399047df658bb7996e" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } } peers { permanent_uuid: "dea7373b963145f6b7749b677830143f" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } }
I20260925 10:12:31.233328 4331 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfdb76636ed644359ebc0b4c24a12c8c, State: Initialized, Role: FOLLOWER
I20260925 10:12:31.233381 4335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.222085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.14.41.126:36419)
0925 10:12:31.222377 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.222378 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.222385 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:12:31.222402 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.222403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.222411 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.222462 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.222991 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.223462 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.224568 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.224844 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.224906 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.230117 (+ 5211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:31.230118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:31.230119 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:31.230126 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:31.231738 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.232160 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.232161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.232161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.232222 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.232438 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.232439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.232439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.232598 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:31.232765 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:31.233174 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 10:12:31.233260 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274}
I20260925 10:12:31.233435 4331 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } } peers { permanent_uuid: "1cff848fecd746399047df658bb7996e" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } } peers { permanent_uuid: "dea7373b963145f6b7749b677830143f" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } }
Sep 25 10:12:31 dist-test-slave-s6tm krb5kdc[4250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331151, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.124@KRBTEST.COM for kudu/127.14.41.124@KRBTEST.COM
I20260925 10:12:31.233812 4331 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:31.233965 4488 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } } peers { permanent_uuid: "1cff848fecd746399047df658bb7996e" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } } peers { permanent_uuid: "dea7373b963145f6b7749b677830143f" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } } }
I20260925 10:12:31.234056 4488 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:12:31.235913 4499 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:12:31.236045 4499 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:12:31.237107 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.229930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:37955 (local address 127.0.0.1:54680)
0925 10:12:31.229961 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:31.229970 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:31.230006 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:31.230138 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:31.230139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:31.230287 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:31.230326 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:31.230328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.231209 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.231210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:31.231679 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:31.231680 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.232201 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.232202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:31.232246 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.232645 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:31.232650 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:31.234052 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:31.236122 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:31.236123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:31.236123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:31.236328 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:31.236537 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:31.236538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:31.236538 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:31.236588 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:31.236842 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:31.236843 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:31.236875 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 10:12:31.236916 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 10:12:31.237007 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260925 10:12:31.237406 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.229963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54680 (local address 127.14.41.124:37955)
0925 10:12:31.230008 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.230010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.230047 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:12:31.230067 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.230067 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.230079 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.230129 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.230386 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.231124 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.231767 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.232100 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.232161 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.234164 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:31.234165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:31.234166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:31.234173 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:31.236039 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.236404 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.236405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.236406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.236472 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.236650 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.236650 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.236651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.236772 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:31.237210 (+ 438us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:31.237249 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 10:12:31.237319 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:12:31.239168 4481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f: Bootstrap starting.
Sep 25 10:12:31 dist-test-slave-s6tm krb5kdc[4250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.125@KRBTEST.COM
I20260925 10:12:31.236732 4263 master_runner.cc:186] Error getting master registration for 127.14.41.125:45017: OK, instance_id { permanent_uuid: "1cff848fecd746399047df658bb7996e" instance_seqno: 1790331151033085 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:12:31.239434 4263 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:12:31.239938 4481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:31.240284 4481 log.cc:824] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:31.241173 4481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f: No bootstrap required, opened a new log
I20260925 10:12:31.243045 4411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.234203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.14.41.125:45017)
0925 10:12:31.234242 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.234245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.234280 (+ 35us) server_negotiation.cc:407] Connection header received
0925 10:12:31.234349 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.234350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.234360 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.234413 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.234914 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.235622 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.236574 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.236858 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.236915 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.239845 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:31.239847 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:31.239847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:31.239855 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:31.241694 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.242106 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.242108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.242108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.242175 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.242402 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.242403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.242403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.242546 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:31.242727 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:31.242768 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 10:12:31.242849 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:12:31.243505 4481 raft_consensus.cc:348] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } } peers { permanent_uuid: "1cff848fecd746399047df658bb7996e" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } } peers { permanent_uuid: "dea7373b963145f6b7749b677830143f" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } }
I20260925 10:12:31.243672 4481 raft_consensus.cc:729] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea7373b963145f6b7749b677830143f, State: Initialized, Role: FOLLOWER
I20260925 10:12:31.243774 4481 consensus_queue.cc:260] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } } peers { permanent_uuid: "1cff848fecd746399047df658bb7996e" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } } peers { permanent_uuid: "dea7373b963145f6b7749b677830143f" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } }
I20260925 10:12:31.244144 4481 sys_catalog.cc:564] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:31.244401 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:31.245491 4305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "dfdb76636ed644359ebc0b4c24a12c8c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45530
I20260925 10:12:31.245656 4305 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 10:12:31.245704 4305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:31.245904 4502 sys_catalog.cc:455] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } } peers { permanent_uuid: "1cff848fecd746399047df658bb7996e" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } } peers { permanent_uuid: "dea7373b963145f6b7749b677830143f" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } } }
I20260925 10:12:31.245992 4502 sys_catalog.cc:458] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:12:31.246709 4513 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 10:12:31.246709 4305 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } } peers { permanent_uuid: "1cff848fecd746399047df658bb7996e" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } } peers { permanent_uuid: "dea7373b963145f6b7749b677830143f" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } }
W20260925 10:12:31.246840 4513 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:12:31.246943 4305 leader_election.cc:288] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [CANDIDATE]: Term 1 election: Requested vote from peers 1cff848fecd746399047df658bb7996e (127.14.41.125:45017), dea7373b963145f6b7749b677830143f (127.14.41.124:37955)
I20260925 10:12:31.247721 4455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dea7373b963145f6b7749b677830143f"
I20260925 10:12:31.247831 4455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:31.248744 4455 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfdb76636ed644359ebc0b4c24a12c8c in term 1.
I20260925 10:12:31.249621 4279 leader_election.cc:302] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dea7373b963145f6b7749b677830143f, dfdb76636ed644359ebc0b4c24a12c8c; no voters:
I20260925 10:12:31.249768 4488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:31.249943 4488 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [term 1 LEADER]: Becoming Leader. State: Replica: dfdb76636ed644359ebc0b4c24a12c8c, State: Running, Role: LEADER
I20260925 10:12:31.250039 4488 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } } peers { permanent_uuid: "1cff848fecd746399047df658bb7996e" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } } peers { permanent_uuid: "dea7373b963145f6b7749b677830143f" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } }
I20260925 10:12:31.250505 4515 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfdb76636ed644359ebc0b4c24a12c8c. Latest consensus state: current_term: 1 leader_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } } peers { permanent_uuid: "1cff848fecd746399047df658bb7996e" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } } peers { permanent_uuid: "dea7373b963145f6b7749b677830143f" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } } }
I20260925 10:12:31.250705 4515 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:31.252718 4413 master_runner.cc:186] Error getting master registration for 127.14.41.125:45017: OK, instance_id { permanent_uuid: "1cff848fecd746399047df658bb7996e" instance_seqno: 1790331151033085 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:12:31.255604 4413 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:12:31.260497 4517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:31.260824 4517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:31.268105 4411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.252156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59484 (local address 127.14.41.125:45017)
0925 10:12:31.252196 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.252198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.256762 (+ 4564us) server_negotiation.cc:407] Connection header received
0925 10:12:31.256868 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.256869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.256880 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.256909 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.257683 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.258565 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.259673 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.260014 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.260058 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.264309 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:31.264311 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:31.264311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:31.264320 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:31.266182 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.266585 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.266586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.266587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.266647 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.267143 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.267144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.267144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.267297 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:31.267641 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:31.267878 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 10:12:31.268008 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:12:31.270040 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.251755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54690 (local address 127.14.41.124:37955)
0925 10:12:31.251795 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.251797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.256209 (+ 4412us) server_negotiation.cc:407] Connection header received
0925 10:12:31.256366 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.256367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.256389 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.256418 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.257979 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.258729 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.261930 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.262266 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.262311 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.266554 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:31.266556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:31.266556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:31.266565 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:31.268318 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.268954 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.268956 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.268956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.269017 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.269285 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.269286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.269286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.269414 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:31.269527 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:31.269844 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 10:12:31.269932 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:12:31.271163 4455 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f [term 1 FOLLOWER]: Refusing update from remote peer dfdb76636ed644359ebc0b4c24a12c8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:31.271592 4515 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dea7373b963145f6b7749b677830143f" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:31.272683 4406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } attempt: 3
I20260925 10:12:31.273690 4502 sys_catalog.cc:455] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfdb76636ed644359ebc0b4c24a12c8c. Latest consensus state: current_term: 1 leader_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } } peers { permanent_uuid: "1cff848fecd746399047df658bb7996e" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } } peers { permanent_uuid: "dea7373b963145f6b7749b677830143f" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } } }
I20260925 10:12:31.273799 4502 sys_catalog.cc:458] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:12:31.274832 4515 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } } peers { permanent_uuid: "1cff848fecd746399047df658bb7996e" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } } peers { permanent_uuid: "dea7373b963145f6b7749b677830143f" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } } }
I20260925 10:12:31.274925 4515 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:31.275607 4525 mvcc.cc:205] Tried to move back new op lower bound from 7333196395594903552 to 7333196395520950272. Current Snapshot: MvccSnapshot[applied={T|T < 7333196395594903552}]
I20260925 10:12:31.275697 4335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.255919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45540 (local address 127.14.41.126:36419)
0925 10:12:31.256007 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.256020 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.261783 (+ 5763us) server_negotiation.cc:407] Connection header received
0925 10:12:31.261881 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.261882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.261892 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.261921 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.262565 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.263370 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.264517 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.264820 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.264859 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.271558 (+ 6699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:31.271559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:31.271560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:31.271582 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:31.273763 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.274464 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.274465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.274466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.274532 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.274880 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.274881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.274881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.275024 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:31.275109 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:31.275453 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 10:12:31.275542 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 10:12:31.276016 4526 mvcc.cc:205] Tried to move back new op lower bound from 7333196395594903552 to 7333196395520950272. Current Snapshot: MvccSnapshot[applied={T|T < 7333196395594903552}]
I20260925 10:12:31.276139 4502 sys_catalog.cc:455] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } } peers { permanent_uuid: "1cff848fecd746399047df658bb7996e" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } } peers { permanent_uuid: "dea7373b963145f6b7749b677830143f" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } } }
I20260925 10:12:31.276260 4502 sys_catalog.cc:458] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:12:31.276757 4517 catalog_manager.cc:1347] Generated new cluster ID: b6ad2f9a51a14dbeba66d7dfbd2262f4
I20260925 10:12:31.276818 4517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:12:31 dist-test-slave-s6tm krb5kdc[4250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331150, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.125@KRBTEST.COM for kudu/127.14.41.124@KRBTEST.COM
I20260925 10:12:31.282269 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.272952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:37955 (local address 127.0.0.1:54700)
0925 10:12:31.273044 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:31.273057 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:31.273097 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:31.273900 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:31.273901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:31.274085 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:31.274144 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:31.274147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.275123 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.275124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:31.275953 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:31.275957 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.276784 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.276785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:31.276864 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.277430 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:31.277436 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:31.279113 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:31.281046 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:31.281048 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:31.281048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:31.281287 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:31.281493 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:31.281493 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:31.281494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:31.281573 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:31.281863 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:31.281865 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:31.281923 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 10:12:31.281992 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 10:12:31.282116 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 10:12:31.282673 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:31.273189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.14.41.124:37955)
0925 10:12:31.273272 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.273286 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.273294 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:12:31.273317 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.273318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.273329 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.273380 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.274249 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.275013 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.276344 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:31.276650 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:31.276712 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:31.279210 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:31.279212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:31.279212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:31.279221 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:31.280949 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.281372 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.281372 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.281373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.281432 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:31.281632 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:31.281633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:31.281633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:31.281761 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:31.281828 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:31.282488 (+ 660us) server_negotiation.cc:299] Negotiation successful
0925 10:12:31.282581 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:12:31.284412 4406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e: Bootstrap starting.
I20260925 10:12:31.285194 4406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:31.285501 4406 log.cc:824] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:31.292776 4406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e: No bootstrap required, opened a new log
I20260925 10:12:31.294898 4406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } } peers { permanent_uuid: "1cff848fecd746399047df658bb7996e" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } } peers { permanent_uuid: "dea7373b963145f6b7749b677830143f" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } }
I20260925 10:12:31.295059 4406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cff848fecd746399047df658bb7996e, State: Initialized, Role: FOLLOWER
I20260925 10:12:31.295143 4406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } } peers { permanent_uuid: "1cff848fecd746399047df658bb7996e" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } } peers { permanent_uuid: "dea7373b963145f6b7749b677830143f" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } }
I20260925 10:12:31.295501 4406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:31.296216 4380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1cff848fecd746399047df658bb7996e"
I20260925 10:12:31.296329 4380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:31.297241 4380 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfdb76636ed644359ebc0b4c24a12c8c in term 1.
I20260925 10:12:31.297725 4379 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e [term 1 FOLLOWER]: Refusing update from remote peer dfdb76636ed644359ebc0b4c24a12c8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:31.298018 4515 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cff848fecd746399047df658bb7996e" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:31.300024 4531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } } peers { permanent_uuid: "1cff848fecd746399047df658bb7996e" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } } peers { permanent_uuid: "dea7373b963145f6b7749b677830143f" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } } }
I20260925 10:12:31.300153 4531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:12:31.300220 4531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfdb76636ed644359ebc0b4c24a12c8c. Latest consensus state: current_term: 1 leader_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } } peers { permanent_uuid: "1cff848fecd746399047df658bb7996e" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } } peers { permanent_uuid: "dea7373b963145f6b7749b677830143f" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } } }
I20260925 10:12:31.300285 4531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:12:31.300323 4531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } } peers { permanent_uuid: "1cff848fecd746399047df658bb7996e" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } } peers { permanent_uuid: "dea7373b963145f6b7749b677830143f" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } } }
I20260925 10:12:31.300381 4531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:12:31.308506 4533 mvcc.cc:205] Tried to move back new op lower bound from 7333196395594903552 to 7333196395520950272. Current Snapshot: MvccSnapshot[applied={T|T < 7333196395594903552}]
I20260925 10:12:31.311018 4488 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfdb76636ed644359ebc0b4c24a12c8c" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 36419 } } peers { permanent_uuid: "1cff848fecd746399047df658bb7996e" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 45017 } } peers { permanent_uuid: "dea7373b963145f6b7749b677830143f" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 37955 } } }
I20260925 10:12:31.311131 4488 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:31.312747 4517 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:31.312812 4546 catalog_manager.cc:1259] Loaded cluster ID: b6ad2f9a51a14dbeba66d7dfbd2262f4
I20260925 10:12:31.312845 4546 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e: loading cluster ID for follower catalog manager: success
W20260925 10:12:31.312935 4546 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260925 10:12:31.312971 4546 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:12:31.313562 4517 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:31.322093 4517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfdb76636ed644359ebc0b4c24a12c8c: Generated new TSK 0
I20260925 10:12:31.322427 4517 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:12:31.352892 4411 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:31.350881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.14.41.125:45017)
0925 10:12:31.350928 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.350930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.352137 (+ 1207us) server_negotiation.cc:407] Connection header received
0925 10:12:31.352260 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.352261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.352274 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.352306 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.352815 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59498: BlockingRecv error: recv got EOF from 127.0.0.1:59498 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 10:12:31.353703 4335 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:31.351456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.14.41.126:36419)
0925 10:12:31.351499 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.351501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.353160 (+ 1659us) server_negotiation.cc:407] Connection header received
0925 10:12:31.353258 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.353259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.353270 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.353300 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.353635 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45552: BlockingRecv error: recv got EOF from 127.0.0.1:45552 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 10:12:31.354482 4486 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:31.351839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54712 (local address 127.14.41.124:37955)
0925 10:12:31.351888 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.351890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.353947 (+ 2057us) server_negotiation.cc:407] Connection header received
0925 10:12:31.354049 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.354050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.354062 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.354092 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.354424 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54712: BlockingRecv error: recv got EOF from 127.0.0.1:54712 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 10:12:31.357362 4411 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:31.356487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59500 (local address 127.14.41.125:45017)
0925 10:12:31.356537 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.356539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.356771 (+ 232us) server_negotiation.cc:407] Connection header received
0925 10:12:31.356891 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.356892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.356913 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.356945 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.357313 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59500: BlockingRecv error: recv got EOF from 127.0.0.1:59500 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 10:12:31.360603 4411 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:31.359688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.14.41.125:45017)
0925 10:12:31.359737 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.359739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.360064 (+ 325us) server_negotiation.cc:407] Connection header received
0925 10:12:31.360167 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.360168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.360180 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.360210 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.360551 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59508: BlockingRecv error: recv got EOF from 127.0.0.1:59508 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260925 10:12:31.364768 4411 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:31.363821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.14.41.125:45017)
0925 10:12:31.363863 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.363865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.364180 (+ 315us) server_negotiation.cc:407] Connection header received
0925 10:12:31.364286 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.364287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.364299 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.364331 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.364710 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59524: BlockingRecv error: recv got EOF from 127.0.0.1:59524 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 10:12:31.371275 4411 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:31.370321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.14.41.125:45017)
0925 10:12:31.370364 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.370366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.370662 (+ 296us) server_negotiation.cc:407] Connection header received
0925 10:12:31.370768 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.370770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.370783 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.370816 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.371214 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59538: BlockingRecv error: recv got EOF from 127.0.0.1:59538 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 10:12:31.381737 4411 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:31.380717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.14.41.125:45017)
0925 10:12:31.380780 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.380783 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.381124 (+ 341us) server_negotiation.cc:407] Connection header received
0925 10:12:31.381231 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.381233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.381245 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.381274 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.381666 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59546: BlockingRecv error: recv got EOF from 127.0.0.1:59546 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260925 10:12:31.400301 4411 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:31.399266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59558 (local address 127.14.41.125:45017)
0925 10:12:31.399313 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.399315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.399620 (+ 305us) server_negotiation.cc:407] Connection header received
0925 10:12:31.399723 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.399724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.399737 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.399767 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.400235 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59558: BlockingRecv error: recv got EOF from 127.0.0.1:59558 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 10:12:31.434506 4411 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:31.433625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59562 (local address 127.14.41.125:45017)
0925 10:12:31.433676 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.433678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.433887 (+ 209us) server_negotiation.cc:407] Connection header received
0925 10:12:31.433917 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.433918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.433932 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.433981 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.434439 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59562: BlockingRecv error: recv got EOF from 127.0.0.1:59562 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 10:12:31.500768 4411 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:31.499844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59564 (local address 127.14.41.125:45017)
0925 10:12:31.499964 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.499966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.500045 (+ 79us) server_negotiation.cc:407] Connection header received
0925 10:12:31.500182 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.500183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.500195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.500243 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.500696 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59564: BlockingRecv error: recv got EOF from 127.0.0.1:59564 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 10:12:31.630856 4411 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:31.629983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59572 (local address 127.14.41.125:45017)
0925 10:12:31.630078 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.630080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.630198 (+ 118us) server_negotiation.cc:407] Connection header received
0925 10:12:31.630310 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.630312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.630326 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.630412 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.630792 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59572: BlockingRecv error: recv got EOF from 127.0.0.1:59572 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 10:12:31.889464 4411 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:31.888421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59586 (local address 127.14.41.125:45017)
0925 10:12:31.888573 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:31.888575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:31.888587 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:12:31.888710 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:31.888711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:31.888726 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:31.888837 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:31.889388 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59586: BlockingRecv error: recv got EOF from 127.0.0.1:59586 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
I20260925 10:12:32.247372 4513 catalog_manager.cc:1259] Loaded cluster ID: b6ad2f9a51a14dbeba66d7dfbd2262f4
I20260925 10:12:32.247452 4513 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f: loading cluster ID for follower catalog manager: success
I20260925 10:12:32.248536 4513 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f: acquiring CA information for follower catalog manager: success
I20260925 10:12:32.248837 4513 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dea7373b963145f6b7749b677830143f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:12:32.314169 4546 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e: acquiring CA information for follower catalog manager: success
I20260925 10:12:32.314505 4546 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1cff848fecd746399047df658bb7996e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:12:32.405591 4613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:32.402677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.14.41.125:45017)
0925 10:12:32.402805 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:32.402807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:32.402908 (+ 101us) server_negotiation.cc:407] Connection header received
0925 10:12:32.402945 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:32.402946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:32.403005 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:32.403103 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:32.403301 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:32.403827 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:32.404707 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:32.404917 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:32.404977 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:32.405258 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:32.405386 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:32.405452 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:32.405499 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 10:12:32.405520 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260925 10:12:32.410326 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:32.407823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56110 (local address 127.14.41.124:37955)
0925 10:12:32.407965 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:32.407967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:32.407976 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:12:32.408039 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:32.408040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:32.408077 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:32.408146 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:32.408225 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:32.408731 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:32.409434 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:32.409656 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:32.409714 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:32.409855 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:32.409963 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:32.410139 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:32.410179 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 10:12:32.410204 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":72,"threads_started":1}
I20260925 10:12:32.415302 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:32.412304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56116 (local address 127.14.41.124:37955)
0925 10:12:32.412353 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:32.412356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:32.412365 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:12:32.412391 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:32.412391 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:32.412404 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:32.412463 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:32.412634 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:32.413442 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:32.414476 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:32.414675 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:32.414709 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:32.414917 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:32.414956 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:32.415005 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:32.415217 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 10:12:32.415241 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:12:32.415293 4613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:32.412307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39906 (local address 127.14.41.125:45017)
0925 10:12:32.412367 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:32.412369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:32.412416 (+ 47us) server_negotiation.cc:407] Connection header received
0925 10:12:32.412485 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:32.412486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:32.412496 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:32.412552 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:32.412685 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:32.413255 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:32.414467 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:32.414764 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:32.414829 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:32.415007 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:32.415038 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:32.415093 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:32.415182 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 10:12:32.415211 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 10:12:32.415546 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:32.412147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50856 (local address 127.14.41.126:36419)
0925 10:12:32.412399 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:32.412401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:32.412410 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:12:32.412603 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:32.412604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:32.412666 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:32.412750 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:32.412895 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:32.413615 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:32.414391 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:32.414715 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:32.414777 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:32.415012 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:32.415164 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:32.415236 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:32.415397 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 10:12:32.415430 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":54,"threads_started":1}
I20260925 10:12:32.417541 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 4263
I20260925 10:12:32.423417 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 4338
I20260925 10:12:32.430610 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 4413
2026-09-25T10:12:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2865 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:32 dist-test-slave-s6tm krb5kdc[4630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:32 dist-test-slave-s6tm krb5kdc[4630](info): set up 2 sockets
Sep 25 10:12:32 dist-test-slave-s6tm krb5kdc[4630](info): commencing operation
krb5kdc: starting...
W20260925 10:12:33.473956 14501 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/test-user.keytab.
Sep 25 10:12:33 dist-test-slave-s6tm krb5kdc[4630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.14.41.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
Entry for principal kudu/127.14.41.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab.
I20260925 10:12:33.607898 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:44669
--webserver_interface=127.14.41.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:43593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:44669,127.14.41.125:37329,127.14.41.124:46119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:33.697274 4643 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:12:33.697513 4643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:33.697531 4643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:33.697553 4643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:33.698670 4643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:33.698719 4643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:33.698732 4643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:33.698745 4643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:33.698757 4643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:33.700578 4643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:43593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:44669,127.14.41.125:37329,127.14.41.124:46119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.126.keytab
--principal=kudu/127.14.41.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.126:44669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.41.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:33.700838 4643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:33.701027 4643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:33.706997 4648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:33.707010 4649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:33.707166 4643 server_base.cc:1034] running on GCE node
W20260925 10:12:33.707012 4651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:33.707898 4643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:33.708297 4643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:33.709487 4643 hybrid_clock.cc:648] HybridClock initialized: now 1790331153709457 us; error 35 us; skew 500 ppm
Sep 25 10:12:33 dist-test-slave-s6tm krb5kdc[4630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331153, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:33.718173 4643 init.cc:377] Logged in from keytab as kudu/127.14.41.126@KRBTEST.COM (short username kudu)
I20260925 10:12:33.718837 4643 webserver.cc:460] Webserver started at http://127.14.41.126:34237/ using document root <none> and password file <none>
I20260925 10:12:33.719192 4643 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:33.719326 4643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:33.719511 4643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:33.721071 4643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-0/data/instance:
uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d"
format_stamp: "Formatted at 2026-09-25 10:12:33 on dist-test-slave-s6tm"
server_key: "ac6fd5920615e4ba0950f2d68bb2e8b9"
server_key_iv: "c8c67d45d000d415826b70c228b5737f"
server_key_version: "encryptionkey@0"
I20260925 10:12:33.721647 4643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-0/wal/instance:
uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d"
format_stamp: "Formatted at 2026-09-25 10:12:33 on dist-test-slave-s6tm"
server_key: "ac6fd5920615e4ba0950f2d68bb2e8b9"
server_key_iv: "c8c67d45d000d415826b70c228b5737f"
server_key_version: "encryptionkey@0"
I20260925 10:12:33.723579 4643 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260925 10:12:33.724996 4658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:33.725162 4643 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:12:33.725229 4643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-0/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-0/wal
uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d"
format_stamp: "Formatted at 2026-09-25 10:12:33 on dist-test-slave-s6tm"
server_key: "ac6fd5920615e4ba0950f2d68bb2e8b9"
server_key_iv: "c8c67d45d000d415826b70c228b5737f"
server_key_version: "encryptionkey@0"
I20260925 10:12:33.725303 4643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:33.751622 4643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:33.754403 4643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:33.754609 4643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:33.758908 4643 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.126:44669
I20260925 10:12:33.759033 4710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.126:44669 every 8 connection(s)
I20260925 10:12:33.759377 4643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:33.760666 4711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:33.762331 4711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:12:33.764038 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 4643
I20260925 10:12:33.764169 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-0/wal/instance
I20260925 10:12:33.764365 14501 external_mini_cluster.cc:1428] Setting key 8645ffb82c3fce90237ad8fca198c293
Sep 25 10:12:33 dist-test-slave-s6tm krb5kdc[4630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331153, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.14.41.125@KRBTEST.COM; defaulting to no policy
I20260925 10:12:33.773968 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:33.763146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:44669 (local address 127.0.0.1:52432)
0925 10:12:33.763320 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:33.763360 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:33.763856 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:33.764226 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:33.764229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:33.764588 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:33.764761 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:33.764766 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:33.766262 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:33.766264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:33.767213 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:33.767215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:33.767751 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:33.767753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:33.767842 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:33.768417 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:33.768428 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:33.770084 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:33.772278 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:33.772284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:33.772286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:33.772479 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:33.772780 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:33.772781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:33.772782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:33.772872 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:33.773354 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:33.773361 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:33.773421 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 10:12:33.773501 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 10:12:33.773797 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":115,"thread_start_us":79,"threads_started":1}
I20260925 10:12:33.773975 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:33.763321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52432 (local address 127.14.41.126:44669)
0925 10:12:33.763463 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:33.763467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:33.763496 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:12:33.764061 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:33.764069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:33.764097 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:33.764202 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:33.764871 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:33.766140 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:33.767296 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:33.767638 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:33.767724 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:33.770176 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:33.770185 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:33.770185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:33.770193 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:33.772194 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:33.772571 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:33.772575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:33.772575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:33.772681 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:33.772955 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:33.772956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:33.772957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:33.773268 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:33.773355 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:33.773573 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 10:12:33.773708 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260925 10:12:33.774853 4711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:12:33.775418 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:33.775215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:37329 (local address 127.0.0.1:51854)
0925 10:12:33.775346 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:33.775396 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.125:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260925 10:12:33.775588 4660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.41.125:37329: connect: Connection refused (error 111)
I20260925 10:12:33.776207 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:33.775989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:37329 (local address 127.0.0.1:51860)
0925 10:12:33.776127 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:33.776178 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.125:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260925 10:12:33.776317 4711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.125:37329: Network error: Client connection negotiation failed: client connection to 127.14.41.125:37329: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.125.keytab.
Entry for principal kudu/127.14.41.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.125.keytab.
I20260925 10:12:33.795038 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.125:37329
--webserver_interface=127.14.41.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.125.keytab
--principal=kudu/127.14.41.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:43593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:44669,127.14.41.125:37329,127.14.41.124:46119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:12:33.825552 4711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } attempt: 1
I20260925 10:12:33.826156 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:33.826004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:37329 (local address 127.0.0.1:51874)
0925 10:12:33.826076 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:33.826122 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.125:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 10:12:33.826529 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:33.826435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:37329 (local address 127.0.0.1:51890)
0925 10:12:33.826479 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:33.826504 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.125:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 10:12:33.826614 4711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.125:37329: Network error: Client connection negotiation failed: client connection to 127.14.41.125:37329: connect: Connection refused (error 111)
W20260925 10:12:33.878645 4718 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:12:33.878871 4718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:33.878888 4718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:33.878911 4718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:33.880124 4718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:33.880169 4718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:33.880183 4718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:33.880204 4718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:33.880261 4718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:33.881762 4718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:43593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:44669,127.14.41.125:37329,127.14.41.124:46119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.125.keytab
--principal=kudu/127.14.41.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.125:37329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.41.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:33.882019 4718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:33.882278 4718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:12:33.888003 4718 server_base.cc:1034] running on GCE node
W20260925 10:12:33.887907 4723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:33.887897 4726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:33.887907 4724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:33.888512 4718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:33.888818 4718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:33.889973 4718 hybrid_clock.cc:648] HybridClock initialized: now 1790331153889966 us; error 35 us; skew 500 ppm
Sep 25 10:12:33 dist-test-slave-s6tm krb5kdc[4630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331153, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:33.892740 4718 init.cc:377] Logged in from keytab as kudu/127.14.41.125@KRBTEST.COM (short username kudu)
I20260925 10:12:33.893271 4718 webserver.cc:460] Webserver started at http://127.14.41.125:42863/ using document root <none> and password file <none>
I20260925 10:12:33.893513 4718 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:33.893582 4718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:33.893738 4718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:33.894778 4711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } attempt: 2
I20260925 10:12:33.894918 4718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-1/data/instance:
uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb"
format_stamp: "Formatted at 2026-09-25 10:12:33 on dist-test-slave-s6tm"
server_key: "023d5d3310da96e0517208ffd3da6404"
server_key_iv: "3131c2cc665584f60be1c31692736062"
server_key_version: "encryptionkey@0"
I20260925 10:12:33.895207 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:33.895059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:37329 (local address 127.0.0.1:51906)
0925 10:12:33.895117 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:33.895169 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.125:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 10:12:33.895368 4718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-1/wal/instance:
uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb"
format_stamp: "Formatted at 2026-09-25 10:12:33 on dist-test-slave-s6tm"
server_key: "023d5d3310da96e0517208ffd3da6404"
server_key_iv: "3131c2cc665584f60be1c31692736062"
server_key_version: "encryptionkey@0"
I20260925 10:12:33.895520 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:33.895423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:37329 (local address 127.0.0.1:51922)
0925 10:12:33.895473 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:33.895498 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.125:37329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 10:12:33.895666 4711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.125:37329: Network error: Client connection negotiation failed: client connection to 127.14.41.125:37329: connect: Connection refused (error 111)
I20260925 10:12:33.897007 4718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:12:33.897966 4733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:33.898248 4718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:12:33.898373 4718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-1/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-1/wal
uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb"
format_stamp: "Formatted at 2026-09-25 10:12:33 on dist-test-slave-s6tm"
server_key: "023d5d3310da96e0517208ffd3da6404"
server_key_iv: "3131c2cc665584f60be1c31692736062"
server_key_version: "encryptionkey@0"
I20260925 10:12:33.898475 4718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:33.926011 4718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:33.928335 4718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:33.928509 4718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:33.933303 4718 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.125:37329
I20260925 10:12:33.933346 4785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.125:37329 every 8 connection(s)
I20260925 10:12:33.933732 4718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-1/data/info.pb
I20260925 10:12:33.934759 4786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:33.936537 4786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:12:33.941306 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 4718
I20260925 10:12:33.941448 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-1/wal/instance
I20260925 10:12:33.941725 14501 external_mini_cluster.cc:1428] Setting key 281777193af0bcca7b5822d5f9f04e2e
Sep 25 10:12:33 dist-test-slave-s6tm krb5kdc[4630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331153, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.125@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:12:33.947911 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:33.937292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52444 (local address 127.14.41.126:44669)
0925 10:12:33.937345 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:33.937350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:33.937574 (+ 224us) server_negotiation.cc:407] Connection header received
0925 10:12:33.938178 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:33.938180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:33.938195 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:33.938266 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:33.938991 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:33.939592 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:33.940833 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:33.941083 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:33.941153 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:33.944383 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:33.944385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:33.944386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:33.944395 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:33.946005 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:33.946597 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:33.946598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:33.946598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:33.946659 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:33.947125 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:33.947126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:33.947127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:33.947253 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:33.947319 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:33.947756 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 10:12:33.947839 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:12:33.948238 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:33.937209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:44669 (local address 127.0.0.1:52444)
0925 10:12:33.937398 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:33.937445 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:33.937929 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:33.938378 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:33.938383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:33.938661 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:33.938878 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:33.938883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:33.939696 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:33.939697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:33.940728 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:33.940730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:33.941225 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:33.941227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:33.941413 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:33.942127 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:33.942138 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:33.944271 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:33.946102 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:33.946114 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:33.946117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:33.946501 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:33.946736 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:33.946738 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:33.946738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:33.946918 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:33.947340 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:33.947349 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:33.947573 (+ 224us) client_negotiation.cc:769] Sending connection context
0925 10:12:33.947715 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 10:12:33.947956 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":106,"threads_started":1}
I20260925 10:12:33.949795 4786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.14.41.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.14.41.124@KRBTEST.COM" created.
Sep 25 10:12:33 dist-test-slave-s6tm krb5kdc[4630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331153, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.125@KRBTEST.COM for kudu/127.14.41.125@KRBTEST.COM
I20260925 10:12:33.959499 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:33.950170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:37329 (local address 127.0.0.1:51936)
0925 10:12:33.950226 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:33.950237 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:33.950303 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:33.950578 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:33.950579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:33.950730 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:33.950779 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:33.950781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:33.952206 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:33.952208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:33.953133 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:33.953138 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:33.953832 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:33.953833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:33.953898 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:33.954416 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:33.954422 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:33.955825 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:33.958295 (+ 2470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:33.958296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:33.958297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:33.958482 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:33.958753 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:33.958754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:33.958754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:33.958822 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:33.959229 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:33.959230 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:33.959273 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 10:12:33.959337 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 10:12:33.959422 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 10:12:33.959663 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:33.950257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.14.41.125:37329)
0925 10:12:33.950439 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:33.950443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:33.950462 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:12:33.950481 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:33.950482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:33.950494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:33.950579 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:33.950863 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:33.952094 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:33.953237 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:33.953741 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:33.953838 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:33.955967 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:33.955979 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:33.955979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:33.955987 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:33.958168 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:33.958582 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:33.958586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:33.958587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:33.958654 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:33.958900 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:33.958901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:33.958902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:33.959145 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:33.959247 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:33.959398 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 10:12:33.959505 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":115,"threads_started":1}
I20260925 10:12:33.960405 4786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:12:33.960845 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:33.960719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:46119 (local address 127.0.0.1:44588)
0925 10:12:33.960773 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:33.960818 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 10:12:33.960973 4736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.41.124:46119: connect: Connection refused (error 111)
I20260925 10:12:33.961455 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:33.961359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:46119 (local address 127.0.0.1:44590)
0925 10:12:33.961412 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:33.961435 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 10:12:33.961561 4786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.124:46119: Network error: Client connection negotiation failed: client connection to 127.14.41.124:46119: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.14.41.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.124.keytab.
Entry for principal kudu/127.14.41.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.124.keytab.
I20260925 10:12:33.972285 14501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskdDYjLr/build/release/bin/kudu
/tmp/dist-test-taskdDYjLr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.124:46119
--webserver_interface=127.14.41.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.124.keytab
--principal=kudu/127.14.41.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.14.41.84:43593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.41.126:44669,127.14.41.125:37329,127.14.41.124:46119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:12:33.986847 4711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } attempt: 3
Sep 25 10:12:33 dist-test-slave-s6tm krb5kdc[4630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331153, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for kudu/127.14.41.125@KRBTEST.COM
I20260925 10:12:33.994441 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:33.987318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:37329 (local address 127.0.0.1:51948)
0925 10:12:33.987392 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:33.987405 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:33.987486 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:33.987680 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:33.987682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:33.987846 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:33.987918 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:33.987920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:33.988647 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:33.988648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:33.989186 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:33.989188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:33.989593 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:33.989595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:33.989673 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:33.990170 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:33.990175 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:33.991638 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:33.993421 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:33.993422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:33.993422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:33.993597 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:33.993821 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:33.993822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:33.993822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:33.993879 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:33.994175 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:33.994176 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:33.994218 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 10:12:33.994276 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 10:12:33.994363 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 10:12:33.994519 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:33.987417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51948 (local address 127.14.41.125:37329)
0925 10:12:33.987495 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:33.987497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:33.987510 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:12:33.987541 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:33.987542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:33.987560 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:33.987662 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:33.988059 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:33.988556 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:33.989276 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:33.989501 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:33.989577 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:33.991738 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:33.991740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:33.991741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:33.991749 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:33.993331 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:33.993676 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:33.993678 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:33.993678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:33.993751 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:33.993942 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:33.993943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:33.993943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:33.994091 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:33.994159 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:33.994329 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 10:12:33.994417 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 10:12:33.995129 4711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:12:33.995568 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:33.995428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:46119 (local address 127.0.0.1:44606)
0925 10:12:33.995492 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:33.995535 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 10:12:33.996073 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:33.995984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:46119 (local address 127.0.0.1:44620)
0925 10:12:33.996018 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:33.996049 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260925 10:12:33.996268 4711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.124:46119: Network error: Client connection negotiation failed: client connection to 127.14.41.124:46119: connect: Connection refused (error 111)
I20260925 10:12:34.010829 4786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } attempt: 1
I20260925 10:12:34.011408 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.011262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:46119 (local address 127.0.0.1:44636)
0925 10:12:34.011333 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:34.011375 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 10:12:34.011835 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.011736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:46119 (local address 127.0.0.1:44650)
0925 10:12:34.011783 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:34.011809 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 10:12:34.011957 4786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.124:46119: Network error: Client connection negotiation failed: client connection to 127.14.41.124:46119: connect: Connection refused (error 111)
I20260925 10:12:34.027472 4711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } attempt: 1
I20260925 10:12:34.028030 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.027848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:46119 (local address 127.0.0.1:44656)
0925 10:12:34.027919 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:34.027986 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 10:12:34.028419 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.028318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:46119 (local address 127.0.0.1:44658)
0925 10:12:34.028368 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:34.028398 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 10:12:34.028574 4711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.124:46119: Network error: Client connection negotiation failed: client connection to 127.14.41.124:46119: connect: Connection refused (error 111)
W20260925 10:12:34.060148 4793 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:12:34.060370 4793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:34.060388 4793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:34.060410 4793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:34.061661 4793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:34.061712 4793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:34.061728 4793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:34.061739 4793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:34.061753 4793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:34.063459 4793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.41.84:43593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.14.41.126:44669,127.14.41.125:37329,127.14.41.124:46119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/krb5kdc/kudu_127.14.41.124.keytab
--principal=kudu/127.14.41.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.41.124:46119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.41.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:34.063750 4793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:34.064042 4793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:34.069401 4799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:34.069571 4801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:34.069418 4798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:34.069427 4793 server_base.cc:1034] running on GCE node
I20260925 10:12:34.069999 4793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:34.070353 4793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:34.071554 4793 hybrid_clock.cc:648] HybridClock initialized: now 1790331154071534 us; error 44 us; skew 500 ppm
Sep 25 10:12:34 dist-test-slave-s6tm krb5kdc[4630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331154, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:34.074723 4793 init.cc:377] Logged in from keytab as kudu/127.14.41.124@KRBTEST.COM (short username kudu)
I20260925 10:12:34.075258 4793 webserver.cc:460] Webserver started at http://127.14.41.124:38121/ using document root <none> and password file <none>
I20260925 10:12:34.075496 4793 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:34.075544 4793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:34.075704 4793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:34.077030 4793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-2/data/instance:
uuid: "bf83e593a98c4717b57dc599d48d112d"
format_stamp: "Formatted at 2026-09-25 10:12:34 on dist-test-slave-s6tm"
server_key: "d1617cfd3639ea7e8265335d7a5ee989"
server_key_iv: "4d001855febae69e0ab3a464ecba0d4a"
server_key_version: "encryptionkey@0"
I20260925 10:12:34.077412 4793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-2/wal/instance:
uuid: "bf83e593a98c4717b57dc599d48d112d"
format_stamp: "Formatted at 2026-09-25 10:12:34 on dist-test-slave-s6tm"
server_key: "d1617cfd3639ea7e8265335d7a5ee989"
server_key_iv: "4d001855febae69e0ab3a464ecba0d4a"
server_key_version: "encryptionkey@0"
I20260925 10:12:34.078933 4793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:12:34.079879 4808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:34.080144 4786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } attempt: 2
I20260925 10:12:34.080201 4793 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:34.080318 4793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-2/data,/tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-2/wal
uuid: "bf83e593a98c4717b57dc599d48d112d"
format_stamp: "Formatted at 2026-09-25 10:12:34 on dist-test-slave-s6tm"
server_key: "d1617cfd3639ea7e8265335d7a5ee989"
server_key_iv: "4d001855febae69e0ab3a464ecba0d4a"
server_key_version: "encryptionkey@0"
I20260925 10:12:34.080404 4793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:34.080617 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.080422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:46119 (local address 127.0.0.1:44668)
0925 10:12:34.080487 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:34.080533 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 10:12:34.080981 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.080864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:46119 (local address 127.0.0.1:44680)
0925 10:12:34.080924 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:34.080957 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 10:12:34.081140 4786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.124:46119: Network error: Client connection negotiation failed: client connection to 127.14.41.124:46119: connect: Connection refused (error 111)
I20260925 10:12:34.103781 4711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } attempt: 2
I20260925 10:12:34.104363 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.104191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:46119 (local address 127.0.0.1:44694)
0925 10:12:34.104278 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:34.104327 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 10:12:34.104812 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.104692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:46119 (local address 127.0.0.1:44704)
0925 10:12:34.104749 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:34.104786 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.41.124:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 10:12:34.104919 4711 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.14.41.124:46119: Network error: Client connection negotiation failed: client connection to 127.14.41.124:46119: connect: Connection refused (error 111)
I20260925 10:12:34.108502 4793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:34.110973 4793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:34.111233 4793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:34.115670 4793 rpc_server.cc:307] RPC server started. Bound to: 127.14.41.124:46119
I20260925 10:12:34.115734 4860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.41.124:46119 every 8 connection(s)
I20260925 10:12:34.116210 4793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-2/data/info.pb
I20260925 10:12:34.117305 4861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:34.117933 14501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskdDYjLr/build/release/bin/kudu as pid 4793
I20260925 10:12:34.118040 14501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskdDYjLr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790330877480195-14501-0/minicluster-data/master-2/wal/instance
I20260925 10:12:34.118239 14501 external_mini_cluster.cc:1428] Setting key fb4b56d71c13c054a84f19775074c3a3
I20260925 10:12:34.119833 4861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:12:34 dist-test-slave-s6tm krb5kdc[4630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.124@KRBTEST.COM
Sep 25 10:12:34 dist-test-slave-s6tm krb5kdc[4630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331154, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.124@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:12:34.129712 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.118781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.14.41.124:46119)
0925 10:12:34.118981 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.118990 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.119016 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:12:34.119517 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.119525 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.119579 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.119702 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.120662 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.121799 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.122569 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.122992 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.123123 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:34.126115 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:34.126124 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:34.126124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:34.126135 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:34.128113 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.128492 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.128495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.128496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.128585 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.128805 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.128806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.128806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.129057 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:34.129211 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:34.129324 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 10:12:34.129468 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
I20260925 10:12:34.129931 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.120796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.126:44669 (local address 127.0.0.1:52454)
0925 10:12:34.121153 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:34.121188 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:34.121253 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:34.121426 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:34.121429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:34.121739 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:34.121821 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:34.121824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.122615 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.122616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:34.123289 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:34.123291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.123730 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.123732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:34.123810 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:34.124403 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:34.124412 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:34.126338 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:34.128631 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:34.128638 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:34.128640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:34.128855 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:34.129113 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:34.129114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:34.129115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:34.129185 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:34.129507 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:34.129514 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:34.129580 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 10:12:34.129684 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 10:12:34.129809 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":139,"threads_started":1}
I20260925 10:12:34.130434 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.120919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.14.41.126:44669)
0925 10:12:34.120985 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.120987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.121258 (+ 271us) server_negotiation.cc:407] Connection header received
0925 10:12:34.121293 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.121295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.121305 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.121369 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.121952 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.122506 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.123387 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.123610 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.123696 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:34.126448 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:34.126450 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:34.126450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:34.126458 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:34.128494 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.128944 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.128945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.128945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.129011 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.129258 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.129259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.129260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.129425 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:34.129501 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:34.129713 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 10:12:34.130295 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 10:12:34.130901 4861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:12:34 dist-test-slave-s6tm krb5kdc[4630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331154, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.124@KRBTEST.COM for kudu/127.14.41.125@KRBTEST.COM
Sep 25 10:12:34 dist-test-slave-s6tm krb5kdc[4630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.126@KRBTEST.COM
I20260925 10:12:34.138612 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.131166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.125:37329 (local address 127.0.0.1:51956)
0925 10:12:34.131230 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:34.131241 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:34.131282 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:34.131405 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:34.131406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:34.131554 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:34.131589 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:34.131591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.132355 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.132356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:34.132883 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:34.132884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.133296 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.133297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:34.133362 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:34.133698 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:34.133702 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:34.135035 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:34.137344 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:34.137346 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:34.137346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:34.137582 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:34.137814 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:34.137814 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:34.137815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:34.137870 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:34.138263 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:34.138264 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:34.138317 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 10:12:34.138399 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 10:12:34.138510 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 10:12:34.138619 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.131256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.14.41.125:37329)
0925 10:12:34.131304 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.131307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.131314 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:12:34.131341 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.131342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.131351 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.131404 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.131672 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.132261 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.132967 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.133207 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.133285 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:34.135152 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:34.135154 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:34.135154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:34.135160 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:34.137253 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.137666 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.137667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.137667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.137741 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.137938 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.137939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.137940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.138159 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:34.138238 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:34.138431 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 10:12:34.138525 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:12:34.138870 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.130718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.14.41.126:44669)
0925 10:12:34.130756 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.130758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.130767 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:12:34.130826 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.130826 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.130838 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.130894 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.131530 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.132064 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.132733 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.132942 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.133014 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:34.135790 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:34.135792 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:34.135792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:34.135800 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:34.137532 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.138002 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.138003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.138003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.138051 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.138256 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.138257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.138257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.138404 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:34.138627 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:34.138677 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 10:12:34.138772 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:12:34.138998 4861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:12:34 dist-test-slave-s6tm krb5kdc[4630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331154, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.124@KRBTEST.COM for kudu/127.14.41.124@KRBTEST.COM
Sep 25 10:12:34 dist-test-slave-s6tm krb5kdc[4630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.14.41.125@KRBTEST.COM
I20260925 10:12:34.146467 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.139310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:46119 (local address 127.0.0.1:44724)
0925 10:12:34.139342 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:34.139352 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:34.139392 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:34.139522 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:34.139523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:34.139687 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:34.139726 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:34.139728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.140679 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.140679 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:34.141269 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:34.141271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.141714 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.141715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:34.141769 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:34.142177 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:34.142182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:34.143533 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:34.145388 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:34.145389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:34.145389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:34.145631 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:34.145865 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:34.145865 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:34.145866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:34.145918 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:34.146202 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:34.146203 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:34.146249 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 10:12:34.146331 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 10:12:34.146403 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 10:12:34.146610 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.139344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44724 (local address 127.14.41.124:46119)
0925 10:12:34.139385 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.139387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.139398 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:12:34.139448 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.139449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.139459 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.139519 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.139811 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.140593 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.141365 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.141621 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.141715 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:34.143622 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:34.143623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:34.143624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:34.143632 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:34.145298 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.145708 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.145709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.145709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.145768 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.145980 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.145981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.145981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.146120 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:34.146210 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:34.146379 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 10:12:34.146465 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:12:34.148130 4861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d: Bootstrap starting.
I20260925 10:12:34.148375 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.140062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51966 (local address 127.14.41.125:37329)
0925 10:12:34.140104 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.140106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.140134 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:12:34.140202 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.140202 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.140210 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.140253 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.140858 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.141287 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.142001 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.142395 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.142467 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:34.145203 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:34.145205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:34.145205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:34.145214 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:34.147121 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.147491 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.147492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.147493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.147549 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.147775 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.147776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.147776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.147895 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:34.148151 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:34.148191 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 10:12:34.148266 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:12:34.149369 4861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:34.149708 4861 log.cc:824] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:34.150048 14501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:34.150508 4861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d: No bootstrap required, opened a new log
I20260925 10:12:34.150997 4685 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52458
I20260925 10:12:34.152933 4861 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } } peers { permanent_uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } } peers { permanent_uuid: "bf83e593a98c4717b57dc599d48d112d" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } }
I20260925 10:12:34.153112 4861 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf83e593a98c4717b57dc599d48d112d, State: Initialized, Role: FOLLOWER
I20260925 10:12:34.153199 4861 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } } peers { permanent_uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } } peers { permanent_uuid: "bf83e593a98c4717b57dc599d48d112d" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } }
I20260925 10:12:34.153584 4861 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:34.153537 4867 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } } peers { permanent_uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } } peers { permanent_uuid: "bf83e593a98c4717b57dc599d48d112d" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } } }
I20260925 10:12:34.153693 4867 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:12:34.155616 4878 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:12:34.155701 4878 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:12:34.156191 4793 master_runner.cc:186] Error getting master registration for 127.14.41.126:44669: OK, instance_id { permanent_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" instance_seqno: 1790331153757610 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:12:34.158883 4793 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:12:34.172322 4786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } attempt: 3
Sep 25 10:12:34 dist-test-slave-s6tm krb5kdc[4630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331153, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.125@KRBTEST.COM for kudu/127.14.41.124@KRBTEST.COM
I20260925 10:12:34.181882 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.172874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44728 (local address 127.14.41.124:46119)
0925 10:12:34.172925 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.172927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.172937 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:12:34.172974 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.172974 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.172983 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.173137 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.173533 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.174376 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.175532 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.175888 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.176013 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:34.178546 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:34.178548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:34.178549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:34.178559 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:34.180330 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.180809 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.180810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.180811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.180876 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.181191 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.181192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.181192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.181419 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:34.181498 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:34.181681 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 10:12:34.181774 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:12:34.181883 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.172753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:46119 (local address 127.0.0.1:44728)
0925 10:12:34.172819 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:34.172833 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:34.172882 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:34.173148 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:34.173150 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:34.173366 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:34.173426 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:34.173428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.174486 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.174487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:34.175417 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:34.175420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.176007 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.176008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:34.176080 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:34.176624 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:34.176631 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:34.178425 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:34.180438 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:34.180439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:34.180439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:34.180708 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:34.180977 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:34.180979 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:34.180979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:34.181095 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:34.181499 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:34.181500 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:34.181560 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 10:12:34.181638 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 10:12:34.181760 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 10:12:34.184058 4786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb: Bootstrap starting.
I20260925 10:12:34.184923 4786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:34.185268 4786 log.cc:824] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:34.186250 4786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb: No bootstrap required, opened a new log
I20260925 10:12:34.188712 4786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } } peers { permanent_uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } } peers { permanent_uuid: "bf83e593a98c4717b57dc599d48d112d" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } }
I20260925 10:12:34.188887 4786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ac426c8ee9d4afaa89a8b4353f3d5bb, State: Initialized, Role: FOLLOWER
I20260925 10:12:34.188974 4786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } } peers { permanent_uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } } peers { permanent_uuid: "bf83e593a98c4717b57dc599d48d112d" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } }
I20260925 10:12:34.189337 4881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } } peers { permanent_uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } } peers { permanent_uuid: "bf83e593a98c4717b57dc599d48d112d" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } } }
I20260925 10:12:34.189386 4786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:34.189460 4881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:12:34.191880 4892 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:12:34.192003 4892 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:12:34.192167 4718 master_runner.cc:186] Error getting master registration for 127.14.41.126:44669: OK, instance_id { permanent_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" instance_seqno: 1790331153757610 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:12:34.195628 4718 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:12:34.204056 4711 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } attempt: 3
Sep 25 10:12:34 dist-test-slave-s6tm krb5kdc[4630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331153, etypes {rep=17 tkt=17 ses=17}, kudu/127.14.41.126@KRBTEST.COM for kudu/127.14.41.124@KRBTEST.COM
I20260925 10:12:34.213186 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.204612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44738 (local address 127.14.41.124:46119)
0925 10:12:34.204659 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.204661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.204675 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:12:34.204705 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.204706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.204717 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.204801 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.205261 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.206063 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.207141 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.207460 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.207536 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:34.210111 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:34.210113 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:34.210113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:34.210122 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:34.211833 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.212323 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.212324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.212325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.212394 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.212595 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.212596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.212597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.212746 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:34.212824 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:34.212995 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 10:12:34.213084 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:12:34.213186 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.204500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.41.124:46119 (local address 127.0.0.1:44738)
0925 10:12:34.204568 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:34.204580 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:34.204627 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:34.204865 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:34.204866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:34.205089 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:34.205149 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:34.205151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.206185 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.206186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:34.207045 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:34.207049 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.207556 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.207558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:34.207624 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:34.208245 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:34.208251 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:34.209995 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:34.211977 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:34.211979 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:34.211979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:34.212237 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:34.212463 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:34.212463 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:34.212464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:34.212529 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:34.212814 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:34.212814 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:34.212878 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 10:12:34.212949 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 10:12:34.213074 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 10:12:34.215473 4711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d: Bootstrap starting.
I20260925 10:12:34.216358 4711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:34.216703 4711 log.cc:824] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:34.217859 4711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d: No bootstrap required, opened a new log
I20260925 10:12:34.219429 4711 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } } peers { permanent_uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } } peers { permanent_uuid: "bf83e593a98c4717b57dc599d48d112d" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } }
I20260925 10:12:34.219591 4711 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2e22387e0d4b0e84de67c5fc6ab87d, State: Initialized, Role: FOLLOWER
I20260925 10:12:34.219738 4711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } } peers { permanent_uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } } peers { permanent_uuid: "bf83e593a98c4717b57dc599d48d112d" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } }
I20260925 10:12:34.220140 4711 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:34.220165 4685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 10:12:34.220239 4685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:34.220746 4895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } } peers { permanent_uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } } peers { permanent_uuid: "bf83e593a98c4717b57dc599d48d112d" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } } }
I20260925 10:12:34.221143 4895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:12:34.221407 4685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } } peers { permanent_uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } } peers { permanent_uuid: "bf83e593a98c4717b57dc599d48d112d" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } }
I20260925 10:12:34.221661 4685 leader_election.cc:288] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [CANDIDATE]: Term 1 election: Requested vote from peers 2ac426c8ee9d4afaa89a8b4353f3d5bb (127.14.41.125:37329), bf83e593a98c4717b57dc599d48d112d (127.14.41.124:46119)
I20260925 10:12:34.222235 4760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb"
I20260925 10:12:34.222383 4760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:34.223315 4760 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e22387e0d4b0e84de67c5fc6ab87d in term 1.
I20260925 10:12:34.223599 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bf83e593a98c4717b57dc599d48d112d"
I20260925 10:12:34.223711 4835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:34.225333 4835 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2e22387e0d4b0e84de67c5fc6ab87d in term 1.
I20260925 10:12:34.225871 4661 leader_election.cc:302] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2e22387e0d4b0e84de67c5fc6ab87d, bf83e593a98c4717b57dc599d48d112d; no voters:
I20260925 10:12:34.226039 4895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:34.226191 4895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [term 1 LEADER]: Becoming Leader. State: Replica: 6a2e22387e0d4b0e84de67c5fc6ab87d, State: Running, Role: LEADER
I20260925 10:12:34.226289 4895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } } peers { permanent_uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } } peers { permanent_uuid: "bf83e593a98c4717b57dc599d48d112d" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } }
I20260925 10:12:34.226711 4895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a2e22387e0d4b0e84de67c5fc6ab87d. Latest consensus state: current_term: 1 leader_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } } peers { permanent_uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } } peers { permanent_uuid: "bf83e593a98c4717b57dc599d48d112d" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } } }
I20260925 10:12:34.226790 4895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [sys.catalog]: This master's current role is: LEADER
W20260925 10:12:34.227974 4916 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:12:34.228117 4916 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:12:34.239773 4910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:34.240058 4910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:34.247994 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.223095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.14.41.126:44669)
0925 10:12:34.223140 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.223142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.224083 (+ 941us) server_negotiation.cc:407] Connection header received
0925 10:12:34.224187 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.224188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.224199 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.224229 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.225211 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.225947 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.233097 (+ 7150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.233399 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.233442 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:34.241194 (+ 7752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:34.241195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:34.241196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:34.241204 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:34.242923 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.244006 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.244008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.244008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.244075 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.244311 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.244312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.244312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.244456 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:34.244541 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:34.247784 (+ 3243us) server_negotiation.cc:299] Negotiation successful
0925 10:12:34.247868 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:12:34.250806 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.223465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44748 (local address 127.14.41.124:46119)
0925 10:12:34.223502 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.223504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.225108 (+ 1604us) server_negotiation.cc:407] Connection header received
0925 10:12:34.226968 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.226969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.226997 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.227029 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.231272 (+ 4243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.232069 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.236014 (+ 3945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.236327 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.236367 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:34.245228 (+ 8861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:34.245230 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:34.245230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:34.245239 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:34.246926 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.249837 (+ 2911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.249839 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.249839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.249911 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.250160 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.250161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.250162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.250299 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:34.250386 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:34.250610 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 10:12:34.250711 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:12:34.252900 4835 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d [term 1 FOLLOWER]: Refusing update from remote peer 6a2e22387e0d4b0e84de67c5fc6ab87d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:34.253268 4905 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf83e593a98c4717b57dc599d48d112d" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:34.254714 4867 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a2e22387e0d4b0e84de67c5fc6ab87d. Latest consensus state: current_term: 1 leader_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } } peers { permanent_uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } } peers { permanent_uuid: "bf83e593a98c4717b57dc599d48d112d" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } } }
I20260925 10:12:34.254820 4867 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:12:34.255079 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:34.223762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51980 (local address 127.14.41.125:37329)
0925 10:12:34.223799 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.223801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.235238 (+ 11437us) server_negotiation.cc:407] Connection header received
0925 10:12:34.235354 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.235355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.235367 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.235397 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.236893 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.237613 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.244744 (+ 7131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:34.245085 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:34.245133 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:34.251641 (+ 6508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:34.251643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:34.251643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:34.251652 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:34.253416 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.253947 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.253949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.253949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.254176 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:34.254414 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:34.254415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:34.254415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:34.254550 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:34.254655 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:34.254885 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 10:12:34.254990 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:12:34.255474 4760 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb [term 1 FOLLOWER]: Refusing update from remote peer 6a2e22387e0d4b0e84de67c5fc6ab87d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260925 10:12:34.255532 4664 tablet.cc:2369] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:12:34.255741 4895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:34.255905 4905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } } peers { permanent_uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } } peers { permanent_uuid: "bf83e593a98c4717b57dc599d48d112d" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } } }
I20260925 10:12:34.256017 4905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:34.256306 4867 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } } peers { permanent_uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } } peers { permanent_uuid: "bf83e593a98c4717b57dc599d48d112d" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } } }
I20260925 10:12:34.256392 4867 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:12:34.256584 4918 mvcc.cc:205] Tried to move back new op lower bound from 7333196407817433088 to 7333196407711617024. Current Snapshot: MvccSnapshot[applied={T|T < 7333196407817433088}]
I20260925 10:12:34.257134 4919 mvcc.cc:205] Tried to move back new op lower bound from 7333196407817433088 to 7333196407711617024. Current Snapshot: MvccSnapshot[applied={T|T < 7333196407817433088}]
I20260925 10:12:34.257201 4881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a2e22387e0d4b0e84de67c5fc6ab87d. Latest consensus state: current_term: 1 leader_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } } peers { permanent_uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } } peers { permanent_uuid: "bf83e593a98c4717b57dc599d48d112d" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } } }
I20260925 10:12:34.257292 4881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:12:34.257349 4881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } } peers { permanent_uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } } peers { permanent_uuid: "bf83e593a98c4717b57dc599d48d112d" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } } }
I20260925 10:12:34.257395 4881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:12:34.258211 4910 catalog_manager.cc:1347] Generated new cluster ID: 996349a6491c41ab9e0bfa7180bd7fb9
I20260925 10:12:34.258273 4910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:34.268993 4922 mvcc.cc:205] Tried to move back new op lower bound from 7333196407817433088 to 7333196407711617024. Current Snapshot: MvccSnapshot[applied={T|T < 7333196407817433088}]
I20260925 10:12:34.277858 4905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2e22387e0d4b0e84de67c5fc6ab87d" member_type: VOTER last_known_addr { host: "127.14.41.126" port: 44669 } } peers { permanent_uuid: "2ac426c8ee9d4afaa89a8b4353f3d5bb" member_type: VOTER last_known_addr { host: "127.14.41.125" port: 37329 } } peers { permanent_uuid: "bf83e593a98c4717b57dc599d48d112d" member_type: VOTER last_known_addr { host: "127.14.41.124" port: 46119 } } }
I20260925 10:12:34.278007 4905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:34.279627 4910 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:34.281484 4910 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:34.286056 4910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a2e22387e0d4b0e84de67c5fc6ab87d: Generated new TSK 0
I20260925 10:12:34.286285 4910 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:12:34.311089 4714 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:34.309194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52476 (local address 127.14.41.126:44669)
0925 10:12:34.309242 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.309244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.310274 (+ 1030us) server_negotiation.cc:407] Connection header received
0925 10:12:34.310386 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.310388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.310402 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.310451 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.310981 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52476: BlockingRecv error: recv got EOF from 127.0.0.1:52476 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 10:12:34.312021 4864 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:34.309619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44758 (local address 127.14.41.124:46119)
0925 10:12:34.309660 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.309662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.311427 (+ 1765us) server_negotiation.cc:407] Connection header received
0925 10:12:34.311530 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.311531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.311544 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.311573 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.311902 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44758: BlockingRecv error: recv got EOF from 127.0.0.1:44758 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 10:12:34.312789 4790 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:34.309974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51988 (local address 127.14.41.125:37329)
0925 10:12:34.310025 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.310027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.312240 (+ 2213us) server_negotiation.cc:407] Connection header received
0925 10:12:34.312339 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.312340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.312351 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.312380 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.312724 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51988: BlockingRecv error: recv got EOF from 127.0.0.1:51988 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 10:12:34.315363 4790 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:34.314556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.14.41.125:37329)
0925 10:12:34.314595 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.314597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.314809 (+ 212us) server_negotiation.cc:407] Connection header received
0925 10:12:34.314921 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.314922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.314943 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.314971 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.315309 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52000: BlockingRecv error: recv got EOF from 127.0.0.1:52000 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 10:12:34.318518 4790 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:34.317714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.14.41.125:37329)
0925 10:12:34.317750 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.317752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.317964 (+ 212us) server_negotiation.cc:407] Connection header received
0925 10:12:34.318073 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.318074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.318085 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.318115 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.318459 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52006: BlockingRecv error: recv got EOF from 127.0.0.1:52006 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 10:12:34.322515 4790 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:34.321675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52022 (local address 127.14.41.125:37329)
0925 10:12:34.321712 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.321713 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.321961 (+ 248us) server_negotiation.cc:407] Connection header received
0925 10:12:34.322077 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.322079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.322093 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.322126 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.322454 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52022: BlockingRecv error: recv got EOF from 127.0.0.1:52022 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 10:12:34.328859 4790 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:34.327905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52024 (local address 127.14.41.125:37329)
0925 10:12:34.327985 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.327987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.328281 (+ 294us) server_negotiation.cc:407] Connection header received
0925 10:12:34.328387 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.328388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.328400 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.328429 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.328801 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52024: BlockingRecv error: recv got EOF from 127.0.0.1:52024 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 10:12:34.339619 4790 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:34.338346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52026 (local address 127.14.41.125:37329)
0925 10:12:34.338397 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.338400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.338773 (+ 373us) server_negotiation.cc:407] Connection header received
0925 10:12:34.338892 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.338894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.338908 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.338940 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.339551 (+ 611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52026: BlockingRecv error: recv got EOF from 127.0.0.1:52026 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 10:12:34.357936 4790 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:34.357050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52030 (local address 127.14.41.125:37329)
0925 10:12:34.357095 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.357097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.357367 (+ 270us) server_negotiation.cc:407] Connection header received
0925 10:12:34.357465 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.357466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.357477 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.357506 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.357877 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52030: BlockingRecv error: recv got EOF from 127.0.0.1:52030 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 10:12:34.392439 4790 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:34.391556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52038 (local address 127.14.41.125:37329)
0925 10:12:34.391609 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.391612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.391787 (+ 175us) server_negotiation.cc:407] Connection header received
0925 10:12:34.391862 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.391863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.391876 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.392011 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.392379 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52038: BlockingRecv error: recv got EOF from 127.0.0.1:52038 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 10:12:34.458379 4790 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:34.457630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52042 (local address 127.14.41.125:37329)
0925 10:12:34.457717 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.457719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.457791 (+ 72us) server_negotiation.cc:407] Connection header received
0925 10:12:34.457840 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.457841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.457853 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.457920 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.458314 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52042: BlockingRecv error: recv got EOF from 127.0.0.1:52042 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 10:12:34.588485 4790 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:34.587669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52044 (local address 127.14.41.125:37329)
0925 10:12:34.587767 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.587769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.587833 (+ 64us) server_negotiation.cc:407] Connection header received
0925 10:12:34.587876 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.587877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.587888 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.588021 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.588402 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52044: BlockingRecv error: recv got EOF from 127.0.0.1:52044 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 10:12:34.846652 4790 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:34.845931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52048 (local address 127.14.41.125:37329)
0925 10:12:34.846110 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:34.846113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:34.846123 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:12:34.846151 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:34.846152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:34.846162 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:34.846225 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:34.846597 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52048: BlockingRecv error: recv got EOF from 127.0.0.1:52048 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
I20260925 10:12:35.156231 4878 catalog_manager.cc:1259] Loaded cluster ID: 996349a6491c41ab9e0bfa7180bd7fb9
I20260925 10:12:35.156322 4878 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d: loading cluster ID for follower catalog manager: success
I20260925 10:12:35.157287 4878 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d: acquiring CA information for follower catalog manager: success
I20260925 10:12:35.157701 4878 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bf83e593a98c4717b57dc599d48d112d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:12:35.192524 4892 catalog_manager.cc:1259] Loaded cluster ID: 996349a6491c41ab9e0bfa7180bd7fb9
I20260925 10:12:35.192615 4892 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb: loading cluster ID for follower catalog manager: success
I20260925 10:12:35.193666 4892 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb: acquiring CA information for follower catalog manager: success
I20260925 10:12:35.194073 4892 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2ac426c8ee9d4afaa89a8b4353f3d5bb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:12:35.363524 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:35.360456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52060 (local address 127.14.41.125:37329)
0925 10:12:35.360609 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:35.360611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:35.360623 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:12:35.360659 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:35.360659 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:35.360733 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:35.360826 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:35.361030 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:35.361602 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:35.362533 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:35.362766 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:35.362857 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:35.363098 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:35.363225 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:35.363325 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:35.363371 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 10:12:35.363395 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":59,"threads_started":1}
I20260925 10:12:35.368822 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:35.365900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44770 (local address 127.14.41.124:46119)
0925 10:12:35.366012 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:35.366014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:35.366022 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:12:35.366061 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:35.366061 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:35.366099 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:35.366175 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:35.366296 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:35.366787 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:35.367611 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:35.367849 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:35.367919 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:35.368222 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:35.368341 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:35.368406 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:35.368685 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 10:12:35.368720 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":58,"threads_started":1}
I20260925 10:12:35.373955 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:35.370896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52076 (local address 127.14.41.125:37329)
0925 10:12:35.370943 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:35.370964 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:35.371097 (+ 133us) server_negotiation.cc:407] Connection header received
0925 10:12:35.371153 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:35.371154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:35.371168 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:35.371217 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:35.371419 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:35.372264 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:35.373163 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:35.373373 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:35.373456 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:35.373656 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:35.373696 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:35.373761 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:35.373822 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 10:12:35.373847 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:12:35.374449 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:35.370952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44786 (local address 127.14.41.124:46119)
0925 10:12:35.371000 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:35.371002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:35.371262 (+ 260us) server_negotiation.cc:407] Connection header received
0925 10:12:35.371326 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:35.371327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:35.371339 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:35.371405 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:35.371580 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:35.372333 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:35.373474 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:35.373772 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:35.373846 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:35.374036 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:35.374067 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:35.374107 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:35.374330 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 10:12:35.374355 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:12:35.374475 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:35.370903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52490 (local address 127.14.41.126:44669)
0925 10:12:35.371153 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:35.371156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:35.371343 (+ 187us) server_negotiation.cc:407] Connection header received
0925 10:12:35.371396 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:35.371397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:35.371452 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:35.371558 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:35.371657 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:35.372264 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:35.373484 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:35.373791 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:35.373846 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:35.374073 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:35.374196 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:35.374255 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:35.374344 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 10:12:35.374374 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":178,"threads_started":1}
I20260925 10:12:35.376636 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 4643
I20260925 10:12:35.382460 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 4718
I20260925 10:12:35.387825 14501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskdDYjLr/build/release/bin/kudu with pid 4793
2026-09-25T10:12:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2957 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5822 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (277915 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST